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Abstract 

The  goal  of  this  research  is  to  suggest  a  framework  for  developing  measures  of 
success  for  corporate  level  Air  Force  acquisition  initiatives.  Because  this  research  is 
exploratory,  it  focuses  on  only  one  initiative;  the  2002  Lighting  Bolt  initiative  “Focus  on 
results,  not  process.”  A  qualitative  method  approach  was  used  to  suggest  a  four  part 
framework.  Through  the  review  of  literature,  common  steps  for  creating  metrics  were 
established  and  recurrent  characteristics  of  good  metrics  were  identified.  Then  interviews 
were  conducted  with  acquisition  practitioners  who  have  experience  with  the  initiative. 
Finally,  those  three  parts  were  applied  to  the  initiative  as  a  case  study  and  metrics 
suggested  as  a  result. 

This  study  gives  Air  Force  leaders  clear,  implementable  metrics  that  can  be  used 
as  measures  of  success  for  the  initiative,  and  provides  recommendations  to  improve  this 
initiative’s  performance  and  that  of  future  corporate  Air  Force  acquisition  initiatives. 

This  study  also  gives  leaders  insight  into  whether  or  not  this  initiative  and  others  like  it 
are  an  appropriate  and  effective  way  to  drive  the  changes  they  are  meant  to  bring  about. 
Finally,  from  a  broader  perspective,  the  framework  used  in  this  study  can  be  used  to 
develop  metrics  for  other  corporate  level  initiatives. 
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DEVELOPMENT  OF  MEASURES  OF  SUCCESS 
FOR  CORPORATE  LEVEL  AIR  FORCE  ACQUISITION  INITIATIVES 


I.  Research  Foundation 


Background 

Almost  since  its  inception  in  1947,  the  Air  Force  has  sought  to  reform  the  way  it 
procures  weapon  systems.  Many  factors  involved  in  the  weapon  system  acquisition 
process  are  external  to  the  Air  Force  and  out  of  its  control  (i.e.,  Congressional  constraints, 
the  pace  of  technology  development,  weapon  system  requirements  constantly  changing  as 
real  world  needs  dictate).  However,  self-imposed  administrative  hurdles  are  an  internal 
factor  that  the  Air  Force  can  change  in  order  to  help  improve  its  procurement  practices. 

To  target  the  elements  of  the  acquisition  process  within  the  Air  Force’s  control, 
the  Air  Force  began  implementing  a  series  of  acquisition  reform  initiatives  in  1995. 

These  initiatives,  referred  to  as  “Lightning  Bolts,”  were  created  in  direct  response  to  Air 
Force  leadership’s  growing  concerns  that  it  takes  too  long  to  put  weapon  systems  in  the 
hands  of  the  warfighters  (Department  of  the  Air  Force,  4  February  2003;  Department  of 
the  Air  Force,  1:10  July  2003).  Collectively,  their  purpose  is  to  serve  as  the  catalyst  by 
which  administrative  changes  are  made  in  Air  Force  business  practices  (Senate  Armed 
Services  Committee,  2002).  However,  little  is  known  about  how  to  gauge  the  success  of 
these  initiatives.  Many  metrics  have  been  suggested  for  gauging  the  success  of 
acquisition  reform  attempts  within  the  Department  of  Defense  (DoD),  but  most  of  the 
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metrics  remain  slated  for  use  only  in  individual  acquisition  program  offices  (Pope, 
1997:75-77).  No  list  of  standard  metrics  exists,  and  there  are  no  generally  applicable  and 
logical  methods  to  measure  the  performance  of  acquisition  reform  initiatives  today 
(Beamon,  1999;  Pope,  1997:75-77).  Accordingly,  this  project  is  designed  to  help  better 
understand  these  initiatives  and  how  to  establish  acquisition  based  measures  of  success. 

Gaining  a  better  understanding  of  the  aspects  of  metric  development,  and 
identifying  the  recurrence  of  those  aspects  among  multiple  authors,  will  enable  this 
research  to  create  a  model  upon  which  acquisition  based  metrics  can  be  built.  That  model 
will  then  be  applied  to  the  2002  acquisition  Lightning  Bolt  initiative  “Focus  on  results, 
not  process”  as  a  case  study. 

Problem  Statement 

Senior  Air  Force  leaders  need  to  know  how  well  the  Lightning  Bolt  2002 
initiatives  are  achieving  their  intended  objectives.  This  research  will  develop  the  means 
to  measure  how  successfully  the  Lightning  Bolt  initiative  “Focus  on  results,  not  process” 
is  achieving  its  objectives.  As  a  result,  this  research  will  create  a  model  for  developing 
measures  of  success  that  can  be  applied  to  larger,  corporate  level  Air  Force  acquisition 
initiatives. 

Methodology 

This  research  uses  a  qualitative  research  method  approach.  Qualitative  data  will 
be  collected  and  analyzed  initially,  followed  by  an  independent  evaluation  of  the 
findings.  The  qualitative  research  will  use  a  case  study  to  develop  measures  of  success 
for  the  initiative  of  interest,  and  evaluate  the  effectiveness  of  that  initiative  in  the 


2 


conclusions  and  recommendations  stage.  As  part  of  future  researeh,  quantitative  data  ean 
help  reinforee  and  eonfirm  the  qualitative  results. 

Outline 

Chapter  two  (Literature  Review)  provides  a  history  of  the  Lightning  Bolt  2002 
aequisition  initiatives,  and  explains  why  metries  are  important  to  use  and  their 
eomposition.  It  also  provides  a  description  of  the  general  steps  to  ereating  metries  and 
the  eharaeteristics  that  good  metries  have.  It  then  diseusses  how  metries  are  applieable  to 
aequisition  reform  initiatives. 

Chapter  three  (Methodology)  deseribes  the  methodology  for  the  study,  ineluding 
the  interview  proeess  used  for  data  eolleetion,  the  spiral  method  that  will  be  used  to 
analyze  the  data,  the  teehniques  used  to  validate  the  analysis,  and  the  methods  used  to 
identify  a  generalized  three  step  proeess  for  ereating  metries  and  the  core  attributes  of 
good  metries. 

Chapter  four  (Data  Analysis)  provides  an  analysis  of  interview  data  eolleeted  and 
an  explanation  of  the  resulting  themes  that  evolved.  Those  results  are  then  tailored  to  the 
“Focus  on  results,  not  proeess”  aequisition  initiative. 

Chapter  five  (Conelusions  and  Reeommendations)  summarizes  the  researeh 
findings,  discusses  data  eolleeted  during  interviews  and  the  eonelusions  reaehed  from  that 
data,  and  provides  implementable  reeommendations.  The  ehapter  also  suggests  areas  for 
future  research  and  study. 
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Impact  of  Study 

This  study  will  give  Air  Force  leaders  clear,  implementable  metrics  that  can  be 
used  as  measures  of  success  for  the  Lightning  Bolt  initiative  of  interest.  This  study  will 
also  give  Air  Force  leaders  insight  into  whether  or  not  this  initiative  is  an  effective  and 
appropriate  way  to  drive  the  changes  it  was  meant  to  bring  about.  In  a  broader 
application,  the  framework  used  in  this  study  can  be  used  to  develop  metrics  for  other 
acquisition  based  corporate  Air  Force  level  initiatives. 
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II,  Literature  Review 


This  chapter  provides  the  foundation  upon  which  metrics  to  assess  the  Lightning 
Bolt  2002  initiative  “Focus  on  results,  not  proeess”  ean  be  developed.  In  turn,  the 
research  results  and  metries  ean  be  generalized  for  any  large  aequisition  based  initiative 
and  thus  fulfill  the  eorporate  purpose  of  the  research.  First,  this  chapter  deseribes  the 
background  and  purpose  of  the  speeifie  initiative  being  examined  along  with  five  others 
that  have  been  developed  to  facilitate  the  United  States  Air  Foree  aequisition 
community’s  improvement  and  transformation  efforts.  Next,  the  chapter  reviews  the 
importance  of  metrics  and  what  metrics  are;  general  steps  involved  in  developing  metrics; 
attributes  of  good  metrics;  and  how  metrics  can  be  applied  to  aequisition  reform 
initiatives.  It  is  through  this  review  that  eommonalities  among  theories  of  metric 
development  will  be  used  to  build  a  colleetion  of  attributes  found  to  be  recurrent  among 
good  metrics.  Then,  a  list  of  metrics,  generated  through  a  series  of  interviews,  will  be 
assessed  against  the  characteristics  of  good  metrics  to  construct  a  set  of  useable  metrics 
for  the  Lightning  Bolt  2002  initiative  entitled  “Foeus  on  results,  not  process.” 

Acquisition  Lightning  Bolt  2002  Initiatives 

On  27  February  2002,  in  an  update  to  the  Senate  Armed  Services  Committee 
(2002:^1 1)  on  the  Air  Foree’s  on-going  aequisition  reform  efforts  and  progress,  the 
Assistant  Secretary  of  the  Air  Foree  for  Aequisition,  Dr.  Marvin  Sambur,  reaffirmed  the 
goal  set  for  the  Air  Force  by  the  President  and  the  Secretary  of  Defense  to  transform  the 
military  and  improve  how  it  does  business.  He  stated  that  the  Air  Force  must  reduce  the 
cycle  times  for  moving  new  teehnology  from  the  laboratory  to  the  battlefield.  At  the 
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same  time,  he  said  that  the  aequisition  eommunity  must  improve  their  ability  to  estimate 
both  eosts  and  sehedules  and  greatly  reduee  the  number  of  program  surprises  that 
undermine  eonfidenee  in  our  programs  and  disrupt  our  progress.  Aequisition 
praetitioners  must  inerease  their  delivery  speed  and  regain  eredibility  with  the  warfighter; 
aequisition  praetitioners  must  deliver  what  they  say  they  are  going  to  deliver  when  they 
say  they  are  going  to  deliver  it. 

In  an  effort  to  address  senior  leaders’  desire  to  improve  speed  and  eredibility,  the 
aequisition  leaders  released  the  most  reeent  round  of  Lightning  Bolt  initiatives  in  2002. 
Similar  sets  of  initiatives  have  been  released  in  groups  of  six  to  ten,  approximately  every 
two  years  sinee  1995.  Two  of  the  six  initiatives  released  in  2002  are  proeess  oriented. 

The  “Foeus  on  results,  not  proeess”  initiative  eneourages  streamlining  existing 
aequisition  proeesses,  ehallenging  those  that  do  not  add  value,  and  getting  rid  of  the 
proeesses  that  do  not  make  sense;  in  turn,  the  initiative  allows  aequisition  praetitioners  to 
keep  their  foeus  on  delivering  eapabilities  that  meet  the  needs  of  warfighters.  The  seeond 
proeess  initiative  is  designed  to  strengthen  eontinuing  proeess  improvements  and 
eommunieation  between  the  government  and  eontraetors  by  ereating  a  knowledge 
pipeline.  (Druyun,  2001;  Senate  Armed  Serviees  Committee,  2002) 

The  other  four  2002  initiatives  are  people  oriented.  The  spiral  development 
initiative  is  designed  to  eneourage  eooperation  between  warfighters  and  aequisition 
praetitioners  during  the  development  and  ineremental  delivery  of  warfighting  eapabilities. 
In  other  words,  it  makes  the  standard  way  of  doing  business  between  the  aequisition 
eommunity  and  warfighters  a  eollaborative  effort  whieh  looks  at  the  entire  eapability  the 
warfighter  needs  and  inerementally  delivers  fully  funetional  portions  of  that  eapability  as 
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funding  levels  will  allow,  until  the  entire  capability  can  be  delivered.  The  “road-block 
buster”  initiative  gives  managers  a  single  point  of  contact  to  help  them  remove 
administrative  and  bureaucratic  stumbling  blocks,  thus  freeing  them  to  be  innovative;  the 
Acquisition  Centers  of  Excellence  (ACE)  were  created  to  be  the  single  point  of  contact 
specified  in  this  initiative.  The  initiative  entitled  “breeding  innovators”  targets  changing 
the  ingrained  culture  of  the  acquisition  workforce  through  acquisition  reform  education. 
Einally,  the  last  initiative  sparked  the  creation  of  a  Program  Executive  Officer  (PEO)  for 
service  contracts  in  order  to  ensure  that  the  Air  Eorce  is  leveraging  its  buying  power  as 
the  portion  of  its  money  spent  on  services  contracts  continues  to  increase.  Table  1  is  a 
summary  of  the  title  and  objective  of  each  of  the  2002  Eightning  Bolts.  (Druyun,  2001; 
Senate  Armed  Services  Committee,  2002) 


Table  1 .  Summary  of  2002  Acquisition  Eightning  Bolt  Initiatives 
(Consolidated  from  Druyun,  2001  and  Senate  Armed  Services  Committee,  2002) 


2002  Lightning  Bolt 

Initiative  Objective 

Eocus  on  results,  not 
process 

Drives  “clean-sheet”  approach  to  acquisitions  by 
streamlining  processes  in  order  to  remove  non  value-added 
steps. 

The  knowledge  pipeline 

Creates  a  “knowledge  pipeline”  with  industry  to  ensure 
continual  communication  and  process  improvements 
among  both  contractors  and  the  government. 

Spirals:  success  in 
increments 

Makes  collaborative  spiral  development  the  preferred 
acquisition  approach  and  requires  collaboration  between 
warfighters  and  the  acquisition  community. 

Road-block  busters 

Erees  managers  to  innovate  and  provide  managers  with  a 
focal  point  to  help  them  remove  bureaucratic  roadblocks. 

Breeding  innovators 

Strives  to  refresh,  revitalize,  and  sustain  the  workforce. 

Program  Executive 
Officer  (PEO)/Service 
contracts 

Obtains  the  best  possible  value  out  of  the  rising  portion  of 
Air  Eorce  procurement  money  spent  on  service  contracts. 
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This  research  will  focus  on  the  Lightning  Bolt  2002  initiative  most  closely  linked 
with  the  corporate  goal  set  for  the  Air  Force  to  improve  speed  and  credibility:  “Focus  on 
results,  not  process”.  This  initiative  drives  a  “clean-sheet”  approach  to  acquisitions  by 
streamlining  processes  in  order  to  remove  non  value-added  steps  (Senate  Armed  Services 
Committee,  2002:1jl4).  In  other  words,  the  initiative  frees  up  the  administrative  hands  of 
acquisition  practitioners  to  allow  them  to  be  as  innovative  as  possible  within  the  confines 
of  the  law.  A  recent  example  of  this  initiative  in  action  is  the  rewriting  of  one  of  the 
regulations  that  governs  weapon  system  procurement  within  the  Air  Force,  and  the  rest  of 
the  Department  of  Defense  (DoD),  the  DoD  5000  series  (i.e.,  DoD  Directive  5000.1, 

DoD  Instruction  5000.2,  and  DoD  5000. 2-R).  The  DoD  5000  series  documents  were 
canceled  on  30  October  2002  by  the  Deputy  Secretary  of  Defense,  Paul  Wolfowitz,  then 
interim  guidance  was  issued;  the  5000  series  documents  have  subsequently  been  revised 
(Wolfowitz,  2002).  DoD  5000.1  was  reissued  (New  DoD  System  Acquisition  Process 
(DoD  5000)).  DoD  5000.2  was  revised;  unlike  the  old  version  which  focused  on  multiple 
superfluous  requirements,  the  new  version  goes  in  depth  into  the  acquisition  model  and 
looks  specifically  at  statutory  requirements  and  required  outcomes  (New  DoD  System 
Acquisition  Process  (DoD  5000)).  DoD  5000. 2-R  is  no  longer  a  mandatory  document, 
but  is  serving  as  the  Interim  DoD  Acquisition  Guidebook  until  the  new  streamlined  DoD 
Acquisition  Guidebook  is  completed;  program  managers  and  decision  authorities  are  now 
empowered  to  decide  what  kind  of  information  is  necessary  to  satisfy  regulatory 
requirements  (Defense  Acquisition  University  Presentation;  Department  of  Defense 
(DoD),  2002;  New  DoD  System  Acquisition  Process  (DoD  5000)). 
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With  the  continued  push  for  acquisition  reform  and  the  goal  of  increased  speed 
and  credibility,  senior  Air  Force  leaders  want  to  know  how  well  the  Lightning  Bolts  are 
actually  doing  their  job.  Currently,  no  metrics  are  in  place  to  measure  their  success.  This 
research  will  create  a  framework  for  building  such  measures  of  success  by  examining 
why  metrics  are  important,  what  metrics  are,  the  steps  involved  in  creating  metrics,  the 
characteristics  that  good  metrics  should  possess,  and  how  metrics  can  be  applied  to 
acquisition  reform  initiatives;  the  framework  will  then  be  applied  to  a  recent  initiative 
and  suggest  metrics  for  it. 

Metrics 

Why  bother  to  develop  metrics  in  the  first  place?  Simply  put:  to  improve 
performance  (Antanitus,  2003:10;  Buchheim,  2000:309;  Rummler  and  Brache, 

1995:135).  Metrics  are  frequently  dubbed  ‘performance  measures’,  meaning  they  tap 
how  well  an  organization  is  performing  (Goett,  2003;  Klapper,  Hamblin,  Hutchison, 
Novak,  and  Vivar,  1999;  Lambert  and  Pohlen,  2001:1;  Milliken,  2001).  The  ultimate 
goal  of  metrics  should  be  “performance”  not  the  measures  themselves  (Milliken,  2001). 
Osborne  and  Gaebler  (1992:147-154)  note  that:  if  the  results  of  that  performance  are  not 
measured,  success  cannot  be  differentiated  from  failure;  if  that  success  cannot  be  seen,  it 
cannot  be  rewarded  and  failure  is  likely  being  rewarded  instead;  and  if  failure  cannot  be 
recognized,  it  cannot  be  corrected.  The  purpose  of  performance  measures,  or  metrics,  is 
not  just  to  examine  how  an  organization  is  performing,  but  to  help  it  perform  better 
(Hammer,  2001:109). 
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Metrics  are  used  to  improve  performance;  furthermore,  properly  structured 
metrics  can  drive  superior  performance.  Keebler  and  others  (1999:13)  surveyed  355 
retailers,  manufacturers,  and  transportation  providers  in  the  United  States  and  conducted 
case  studies  of  roughly  two  dozen  companies,  and  discovered  a  great  disparity  in  levels 
of  performance.  The  singularly  most  important  factor  that  Keebler  and  others  (1999: 13) 
found  to  be  driving  superior  performance  was  the  presence  of  well-utilized  and  properly 
structured  measurement  programs.  Inadequately  structured  metrics  can  drive  the  wrong 
behaviors  and  even  result  in  dysfunctional  behaviors  (Neely,  Richards,  Mills,  Platts,  and 
Bourne,  1997).  Even  though  there  has  been  a  high  level  of  interest  in  metrics  within 
industry  and  academia,  and  many  methods  have  been  suggested  to  develop  metrics,  no 
one  has  addressed  what  makes  up  a  well-designed  metric  and  no  one  has  compared  these 
methods  for  effectiveness  (Neely  and  others,  1997).  This  research  proposes  that  a  well- 
designed  metric  is  one  that  is  systematically  created  and  one  that  possesses  the  attributes 
of  good  metrics  found  to  be  common  within  the  literature. 

Additional  evidence  of  how  the  use  of  metrics  has  been  empirically  shown  to 
improve  performance  is  seen  through  the  implementation  of  goal  setting  (see,  for 
example,  Locke,  Shaw,  Saari,  and  Latham  (1981:126)).  Metrics  are  the  feedback 
mechanism  by  which  progress  toward  organizational  goals  is  measured  (Office  of  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  1999:133).  Lor 
example,  as  part  of  a  recent  policy  directive  from  the  Secretary  and  Chief  of  Staff  of  the 
Air  Lorce  on  improving  speed  and  credibility  within  the  acquisition  workforce,  the 
Commander’s  Initial  Guidance  section  states  that  the  overall  goal  is  to  shorten  the  time  it 
takes  for  decisions  and  getting  more  capable  weapon  systems  out  to  the  warfighter  by  a 
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factor  of  four  (Department  of  the  Air  Foree,  10  July  2003:3).  Correspondingly,  the 
metrie  to  determine  if  that  goal  is  met  will  be  eyele  time.  By  virtue  of  a  goal  being  set, 
metries  ean  be  used;  therefore,  based  on  the  literature,  this  researeh  shows  that  empirieal 
evidenee  supports  that  the  use  of  metrics  improves  performanee. 

The  following  examples  further  support  that  the  implementation  of  metries 
inereases  performanee.  The  eoneept  of  assigning  workers  a  specifie  task  (a  term  whieh 
Loeke  (1982:16)  notes  is  basically  the  same  as  a  goal),  along  with  ineentive  pay  and  time 
and  motion  study,  served  as  the  basis  for  Frederiek  W.  Taylor’s  prineiples  of  seientifie 
management  (Latham  and  Loeke,  1979:69;  Taylor,  2001:61-72).  Taylor  inereased  blue 
collar  worker  produetivity  through  the  use  of  his  seientifie  management  system  (Taylor, 
2001:64).  Locke  (1968:157)  later  theorized  that  goal  setting  is  directly  tied  to  task 
performanee;  he  explains  that  diffieult  goals  result  in  higher  performanee  than  easy  goals, 
and  that  speeifie,  hard  goals  lead  to  even  higher  performanee  levels  eompared  to 
generalized  “do  your  best”  goals  or  no  goals  at  all.  Latham  and  Yukl’s  (1975:824-843) 
evaluation  of  Loeke’s  theory  and  their  meta-analysis  of  twenty  seven  published  and 
unpublished  field  researeh  reports  eonsistently  found  that  goal  setting  produees  increased 
performanee.  Also,  Latham  and  Loeke’s  (1979:68-80)  laboratory  and  field  researeh 
showed  that  the  use  of  goal  setting  inereased  produetion  by  an  average  of  nineteen 
pereent.  In  addition,  Loeke  and  others’  (1981:125,131)  evaluation  of  one  hundred  and 
ten  laboratory  and  field  studies  on  goal  setting  effeets  on  task  performanee  found  that 
ninety-nine  of  those  studies  showed  higher  performanee  resulted  from  setting  speeifie, 
hard  goals  than  from  no,  “do  your  best,”  easy,  or  medium  goals.  A  significant  amount  of 
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data  attests  to  the  presenee  of  inereased  performanee  when  goals  are  set,  and,  therefore, 
when  metrics  are  used. 

In  addition  to  metrics’  importance  because  of  improved  performance,  Keebler, 
Mandrodt,  Durtsche,  and  Ledyard  (1999:80-81)  point  out  that  measures  aid  companies  in 
determining  how  to  remain  competitive  and  confirm  the  value  customers  place  on  their 
services.  And  the  underlying  truth  within  the  axiom  that  what  gets  measured  gets 
attention  is  yet  another  reason  to  use  metrics  (Eccles,  1991:131;  Osborne  and  Gaebler, 
1992:146). 

Now  that  the  importance  of  metrics  has  been  discussed,  a  description  of  what 
constitutes  a  metric  will  help  facilitate  the  explanation  of  this  research.  The  Metrics 
Handbook  developed  by  Air  Force  Systems  Command  (1991:1-1)  defines  metrics  as 
meaningful  measures,  and  data  are  meaningful  when  they  allow  action  to  be  taken. 
Similarly,  Antanitus  (2003:1 1)  calls  metrics  items  you  would  like  to  measure.  Metrics 
emphasize  the  customer,  support  organizational  objectives  and  goals,  facilitate  process 
understanding,  and  encourage  continual  improvement  of  how  business  is  done  (AFSC, 
1991:1-1). 

According  to  Clark  and  Wheelwright  (1994:262),  there  are  two  types  of  measures: 
results  measures,  which  tell  a  team  where  it  currently  stands  in  its  attempt  to  reach  a  goal, 
rather  than  how  it  got  there  or  what  it  could  do  differently;  and  process  measures  which 
look  at  activities  and  tasks  within  an  organization  that  produce  given  results.  Also, 
metrics  can  be  expressed  both  qualitatively  and  quantitatively  (Beamon,  1999). 
Quantitative  metrics  are  frequently  preferred  because  qualitative  metrics,  like  “poor,” 
“fair,”  and  “good,”  are  vague  and  hard  to  use  in  a  meaningful  way  (Beamon,  1999). 
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However,  quantitative  metrics  may  not  adequately  discuss  a  system’s  performance  and, 
as  a  result,  may  be  just  as  vague  (Beamon,  1999).  Locke  (1978:600)  points  out  that  it 
should  not  be  assumed  that  specific  quantitative  goals,  and,  in  turn,  metrics,  are 
inevitably  beneficial.  Some  areas  where  results  are  more  difficult  to  measure  may  require 
qualitative  goals,  and,  in  turn,  qualitative  metrics  (Locke,  1978:600).  The  decision 
between  qualitative  and  quantitative  metrics  depends  upon  the  nature  of  the  system  for 
which  the  metrics,  or  goals,  are  being  established. 

It  is  worthwhile  to  note  that  several  concepts  have  become  prevalent  within  the 
business  arena  over  the  last  two  decades  and  have  popularized,  and  somewhat 
revolutionized,  the  use  of  metrics,  namely  the  balanced  scorecard  (Kaplan  and  Norton, 
March  2001)  and  benchmarking  (Eccles,  1991:131).  Kaplan  and  Norton  (March  2001; 
September  2001)  created  a  way  of  linking  metrics  with  the  elements  of  an  organization’s 
strategy  to  create  a  new  strategic  management  system.  Metrics  are  one  component  of  the 
balanced  scorecard.  Benchmarking  is  information  gathering  in  industry  to  compare  an 
organization’s  performance  with  that  of  other  leading  organizations  that  do  the  same  or 
similar  tasks  (Camp,  1989:xiii;  Eccles,  1991:133).  Metrics  are  used  in  benchmarking,  but 
are  not  synonymous  with  it. 

Metrics  have  been  utilized  in  many  ways  to  improve  performance  and  multiple 
methods  have  been  proposed  to  develop  them.  This  research  looks  at  several  of  those 
methods  and  uses  their  similarities  to  propose  one  generally  applicable  three  step  method 
for  creating  metrics. 
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Steps  to  Create  Metrics. 

By  comparing  the  numerous  methods  for  systematically  developing  metrics  that 
exist  within  the  literature,  this  research  found  that  nearly  all  of  the  methods  share  three 
common  steps  which  will  later  be  discussed.  Of  the  literature  reviewed,  twelve  authors 
presented  thirteen  general  frameworks  for  creating  metrics.  The  number  of  steps 
involved  in  each  framework  ranged  from  three  steps  up  to  eleven  steps.  For  example, 
Clark  and  Wheelwright  (1994:272-273)  suggest  a  four  step  method:  a)  define  factors 
critical  to  customer  satisfaction;  b)  map  cross-functional  process  through  which  results 
are  obtained;  c)  identify  capabilities  and  tasks  necessary  to  complete  process 
successfully;  and  d)  design  measures  to  track  those  capabilities  and  tasks.  Rummler  and 
Brache  (1995:137-138)  recommend  a  similar  four  step  sequence:  a)  clearly  establish  the 
most  important  outputs  of  the  process,  job,  or  organization;  b)  for  each  output,  establish 
the  “critical  dimensions”  of  performance;  c)  create  measures  for  every  critical  dimension; 
and  d)  create  standards,  or  goals,  for  each  measure.  In  contrast,  Eccles  and  Pybum 
(1992)  suggest  a  five  step  process  that  does  not  share  the  three  steps  found  to  be  common 
among  the  other  authors:  a)  choose  non-financial  measures  that  will  compliment 
financial  measures,  determine  relationships  between  them,  and  the  create  firm’s  business 
performance  model;  b)  establish  methodology  to  be  used  to  take  the  measures;  c)  select 
the  frequency  and  layout  of  performance  measurement  reports;  d)  adjust  how  personnel 
are  compensated  and  evaluated  to  encourage  desired  behavioral  changes  that  will 
improve  activity  performance;  and  e)  realize  that  a  key  element  of  a  performance 
measurement  system  is  that  it  will  evolve  with  time  as  managers  grow  and  increase  their 
knowledge  of  measures’  relationships  to  one  another  and  as  conditions  change.  A 
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complete  illustration  of  the  general  steps  for  ereating  metries  for  all  twelve  authors  found 
in  the  literature  is  shown  in  Table  2. 
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Table  2.  General  Steps  for  Creating  Metrics  within  Literature 


Step 

AFSC 

Antanitus  (2003) 

Bnnvii(1996) 

Buchheim 

1 

***  Identify  purpose 

***  Map  corp  competencies  (i.e, 
define  and  agree  upon  them)  by 
organization,  division,  department, 
etc.  to  lowest  desired  level 

*♦*  Prepare  documents  to  guide 
organization  and  where  you  want 
to  go  (e  g.,  missioa  vision,  values) 

Involve  all  affected  organizations 
in  development  ofpetfoimance 
metrics 

2 

**  Develop  operational  definition 
starting  with  customer  -  define 
who,  what,  when,  where,  why,  and 
how  of  metric 

**  Map  processes  used  to  carry 
out  your  mission  to  the  core 
competencies 

Do  situation  analysis  to  research 
strengths  and  weaknesses  of 
organization  and  those  of  main 
competitors.  This  is  done  to 
investigate  factors  that  may  help  or 
hinder  reaching  your  goal. 

**  Flow  chart  applicable  process  & 
determine  important  areas  to 
customer  -  identifies  critical 

activities 

3 

Identify  and  examine  existing 
measurement  systems 

*  Identify  metrics  to  be  used  to 
assess  how  well  processes  are 
being  executed 

**  Define  key  business 
fundamentals  and  success  factors. 
Most  critical  and  difficult  step. 

*  Establish  performance 
measurements  and  goals  or 
standards,  and  assign  tasks  of  data 
collection,  analysis,  and  reporting 

4 

*  Create  new  metrics  if  existing 
metrics  are  inadequate 

*  Identifymacroperfoimance 
measures:  a)  identify  measurement 
categories,  b)  measures  within  each 
category  are  brainstormed,  and  c) 
measures  widdled  down  to  vital 

few 

Analyze  and  report  metrics 

5 

Rate  metric  against  attributes  of  a 
"good"  metric 

Measurement  plan  developed 

Compare  actual  performance  with 
goal  or  standard 

6 

Select  appropriate  measurement 
tools 

Create  procedures  andinstmments 
to  collect  data.  Most  time 
consuming  step. 

Evaluate  causes  of  variance  and 
possible  corrective  action 

7 

Baseline  your  process 

8 

Collect  and  analyze  metrics  over 
time 

9 

Finalize  metric  for  external 
presentation 

10 

Initiate  process  improvement 
activities 
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Table  2.  General  Steps  for  Creating  Metrics  within  Literature  (continued) 


Step 

Keebler  and  others  (1999) 

Mentzer  and  Konrad  (1 99 1 ) 

Pinker,  Smith,  and  Booher  (1997) 

Rrtmmler  and  Brachc  (1995) 

1 

**♦  Start  with  organization 
strategy 

***  Determine  item  to  be 
evaluated  (i.e.,  the  goal  or  problem) 
and  its  context 

*  *  *  Cle  arly  identify  metric  ’  s 
purpose 

*  *  *Cle  arly  e  stablish  mo  St 
important  outputs  of  process,  job, 
or  organization 

2 

Understand  customer  needs 

**  Identify  inputs  and  outputs  of 
task  firm  wants  to  evaluate 

Operationally  define  metric 

For  each  output,  establish 
“critical  dimensions”  of 
performance 

3 

Understand  other  measure  drivers 

Analyze  how  measures  are 
obtained,  scmtinize  reliability, 
validity,  and  meaningfulness  of 
existing  scales 

Do  not  “reinvent  the  wheel”; 
review  existing  metrics  to  see  if 
could  be  adapted  to  fulfill  metric’s 
operational  definition 

Create  measures  for  every  critical 
dimension 

4 

T ake  process,  or  big  picture,  view 
of  logistics 

*  Replace  any  unsatisfactory 
measures  with  measures  that  meet 

these  same  criteria 

*  Develop  new  metrics 

Create  standards,  or  goals,  for  each 
measure  (goal  is  expected  level  of 
perfonnance) 

5 

**  Focus  on  key  activities  and 
processes 

Do  cost-benefit  analysis  of 
satisfactory  measures  to  assess 
measure's  cost  versus  usefulness 

Check  metrics’ “goodness  of  fit”; 
determine  if  new  metrics  possess 
attributes  of  goodmetrics 

6 

Assign  process  owner 

Select  method  of  presentation  for 
metric 

7 

*  Use  a  few  key  measures  only 

Conduct  dry  runs  with  metrics,  ask 
if  metrics  measure  what  they  were 
intended  to  measure,  and  ask 
acquisition  practitioners  if  metrics 
are  meaningful;  perform  a  “sanity” 
check 

8 

Stay  away  from  subjective 

measures 

Get  buy-in  or  consensus  from 
participants 

9 

Do  not  use  averaging  excessively 

Develop  database  for  tracking 
metric  data  overtime  and  examining 
trends 

10 

Be  able  to  understand  the  statistics 

Openly  communicate  metrics  and 
adjust  them  based  on  participant 
critiques 

11 

Use  metrics 
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No  empirical  evidence  was  found  within  the  literature  to  suggest  that  any  one 
particular  method  was  better  to  use  than  any  other.  One  size  does  not  fit  all  and  many 
differences  exist  among  the  authors’  approaches,  but  three  basic  steps  remained  common 
among  eleven  of  the  thirteen  frameworks  examined  (INCOSE,  1998:9).  First,  establish  a 
starting  point  upon  which  to  base  the  metrics;  determine  what  you  want  to  measure. 
Second,  identify  the  most  important  elements  of  what  you  want  to  measure.  Third,  create 
specific  metrics  for  those  critical  elements  so  as  to  improve  the  performance  of  the  item 
being  measured.  If  metrics  are  created  by  systematically  following  these  three  general 
steps  and  they  possess  the  attributes  of  good  metrics  they  will  be  properly-structured 
metrics  and  will  have  the  potential  to  drive  superior  performance  (Keebler  and  others, 
1999:13). 

Attributes  of  Good  Metrics. 

Certain  characteristics  distinguish  good  metrics  from  bad  ones  and  well-designed 
metrics  possess  those  good  characteristics.  Fourteen  authors  in  the  literature  describe 
forty  three  distinct  attributes  that  good  metrics  possess.  The  following  are  a 
representative  list  of  the  good  metric  attributes  in  the  literature.  Beamon  (1999)  says  that 
good  metrics  have  six  characteristics:  consistency  with  organizational  goals, 
inclusiveness  of  pertinent  aspects,  measurability,  meeting  of  customer  goals  and  values, 
relating  to  strategic  goals  and  mission  of  organization,  and  universality.  In  comparison, 
Buchheim  (2000:31 1)  describes  good  metrics  as  having  eight  characteristics,  only  one  of 
which  is  common  with  those  cited  by  Beamon  (i.e,  relating  to  strategic  goals  and 
mission).  According  to  Buchheim  (2000:31 1),  good  metrics:  have  a  defined  sensor  that 
gathers  and  records  data,  like  an  automated  test  station  data  file  or  a  clerk;  have  a  defined 
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unit  of  measurement  (e.g.,  hours  per  widget  produced);  are  meaningful  to  the  customer; 
measure  results  versus  process  (e.g.,  measure  the  level  of  skill  demonstrated  using  a 
widget  versus  the  number  of  days  spent  attending  training  sessions);  have  a  regular 
frequency  with  which  reports  and  measurements  are  done  (e.g.,  monthly  average  failure 
rate);  are  simple  to  use;  and  are  understandable.  A  third  source  finds  that  good  metrics 
possess  three  characteristics,  only  one  of  which  is  shared  with  Beamon’s  and  Buchheim’s 
attributes;  Evans  and  Lindsay  (2002:464,466)  agree  with  both  Beamon  and  Buchheim 
that  good  metrics  relate  to  the  strategic  goals  and  mission  of  the  organization  involved, 
but  also  state  that  good  metrics  are  actionable  and  useful. 

Table  3  summarizes  the  attributes  of  “good”  metrics  as  described  in  the  literature. 
The  elements  that  are  common  among  the  research  are  illustrated  with  this  presentation. 
Of  the  works  shown  in  Table  3,  six  authors  claim  that  metrics  should  relate  to  the 
organizational  mission  and  strategic  goals,  five  suggest  that  simplicity  is  an  important 
quality  of  metrics,  and  five  state  that  good  metrics  are  meaningful  to  customers.  Four 
authors  also  point  out  that  metrics  should  be  understandable  and  derivable  from 
economically  collectible  data  (i.e.,  cost  effective).  All  other  attributes  in  Table  3  are 
common  among  three  authors  or  less. 
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Table  3.  Common  Attributes  of  Good  Metrics 
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Keebler  Mentzer  Pinker 


Table  3.  Common  Attributes  of  Good  Metries  (eontinued) 
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Application  of  Metrics  to  Acquisition  Reform  Initiatives. 

In  order  to  create  a  framework  for  developing  measures  of  sueeess  for  eorporate 
level  Air  Foree  aequisition  initiatives,  not  only  must  it  be  understood  how  to  ereate  well- 
struetured  metries,  but  it  must  be  understood  how  those  metries  ean  be  applied  to 
aequisition  reform  initiatives.  Metrics  can  be  used  to  traek  the  progress  of  supply  chain 
initiatives  (Klapper  and  others,  1999).  The  supply  chain  is  made  up  of  all  the  aetivities 
relating  to  the  transformation  and  flow  of  goods  from  the  point  of  extraetion  of  raw 
materials,  through  to  the  end  users  (Monczka  and  others,  2002:4).  The  military 
aequisition  eommunity  manages  and  oversees  the  aetivities  involved  in  the  procurement 
of  weapon  systems,  from  initial  development  and  proeurement,  through  delivery  to  the 
warfighters,  all  the  way  to  the  end  of  a  weapon  system’s  life  cyele  when  it  is  retired  or 
sent  to  the  “bone  yard”  at  Davis-Monthan  Air  Foree  Base,  Arizona;  therefore,  a  supply 
ehain  perspective  is  appropriate  for  analyzing  the  weapon  system  aequisition  and 
management  proeess.  Accordingly,  metrics  can  be  used  to  monitor  the  progress  of 
aequisition  reform  initiatives. 

Metrics  appropriate  for  aequisition  reform  enable  an  organization  to  assess  reform 
initiatives’  effeetiveness  and  implementation  on  both  aequisition  programs  and  the 
aequisition  reform  proeess  itself  (Pope,  1997:75-76).  Groups  within  the  DoD  have 
proposed  various  metries  to  measure  aequisition  reform;  for  instanee.  Pope  (1997:57,75- 
77)  notes  that  the  Navy  was  working  towards  using  metries,  sueh  as  the  average  eycle 
time  for  issuanee  of  requests  for  proposals,  to  help  gauge  the  use  of  some  of  their 
aequisition  reform  initiatives.  But  most  aequisition  metries  have  been  used  for  individual 
aequisition  programs;  for  example,  a  speeific  aequisition  program,  like  a  program  to  buy 
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radios  for  an  aircraft,  might  employ  a  metric  like  schedule  variance  to  measure  the 
percentage  increase  or  decrease  in  the  time  it  takes  a  contractor  to  deliver  radios  bought 
in  one  month  compared  to  the  delivery  time  of  radios  bought  another  month  (Pope, 
1997:75-77).  No  systematic  approach  to  performance  measurement  or  standardized  set 
of  metrics  for  acquisition  reform  initiatives  currently  exists  (Beamon,  1999;  Pope, 
1997:75-77). 

In  an  effort  to  address  the  lack  of  standardized  metrics  for  acquisition  initiatives, 
the  Acquisition  Reform  Benchmarking  Group  (ARBG)  was  established  by  the  DoD  in 
1996  to  help  measure  progress  within  the  arena  of  acquisition  reform  (Pope,  1997:34-35). 
Pope  (1997:35)  determined  that  the  findings  of  the  ARBG  divide  metrics  into  three 
levels:  program,  subordinate,  and  enterprise.  Metrics  at  their  most  basic  level  measure 
elements  within  individual  acquisition  programs,  or  “little  ‘a’”  acquisitions,  as  Sambur 
refers  to  acquisitions  at  the  program  level  (DiCicco,  2003).  Subordinate  metrics  measure 
factors  that  feed  into  the  highest  level  of  metrics,  which  are  enterprise  metrics.  Enterprise 
metrics  measure  the  efficiency  of  overarching  or  generalizable  processes  that  should  be 
measured  across  the  whole  Air  Force  (Pope,  1997:34-35)  (i.e.,  termed  “big  ‘A’” 
Acquisitions  related  metrics  within  this  occupational  community)  (DiCicco,  2003). 
Enterprise  metrics  include  cost,  schedule,  performance,  and  training  metrics.  The 
acquisition  initiative  this  research  focuses  on  pertains  to  “big  ‘A’”  Acquisitions  and  the 
metrics  this  research  will  recommend  be  used  to  assess  that  initiative  are  enterprise  level 
metrics. 

Pope  (1997:26)  found  that  metrics  can  also  be  categorized  by  the  three  types  of 
activities  that  they  measure,  as  defined  by  the  1995  Process  Action  Team  (PAT)  for 
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contract  administration  reform:  go/no-go,  activities,  and  behavioral  changes.  Go/no-go 
metries  show  whether  or  not  an  aetivity  has  taken  place.  Aetivity  metries  illustrate  how 
extensively  an  action  is  occurring.  And  behavioral  ehange  metries  assess  whether  aetions 
are  ereating  the  desired  ehange  in  behavior  or  results.  Part  of  the  objeetive  of  this 
research  is  to  help  determine  whether  or  not  the  use  of  the  aequisition  initiative  of  interest 
is  an  effeetive  way  to  bring  about  the  desired  ehanges  in  the  aequisition  praetitioners’ 
behavior.  Speeifically,  the  aequisition  practitioners  who  partieipated  in  this  researeh 
were  queried  about  whether  the  initiative  of  interest  was  the  most  effeetive  way  to  get 
them  to  take  the  “elean-sheet”  approach  to  acquisitions  that  the  initiative  was  meant  to 
eneourage.  Additionally,  part  of  the  metrics  recommended  by  this  researeh  will  fall 
within  the  behavioral  ehange  eategory.  Now  that  a  framework  for  building  metries  has 
been  established  by  reviewing  metries’  importanee,  eomposition,  ereative  steps,  good 
attributes,  and  application  to  acquisition  reform  initiatives,  that  framework  will  be 
applied  in  a  ease  study. 

Summary 

Aequisition  reform  initiatives  have  been  born  out  of  senior  Air  Foree  leaders’ 
vision  of  a  faster,  better  aequisition  proeess  and  an  improved  relationship  between  the 
aequisition  eommunity  and  the  warfighters  they  support.  This  researeh  foeuses  on 
developing  metrics  for  one  sueh  initiative.  Many  theories  exist  about  what  eonstitutes 
good  metrics  and  what  the  steps  involved  in  ereating  metrics  are.  A  large  part  of  the 
literature  review  and  additional  researeh  for  this  thesis  identify  those  theories  and 
examine  the  eommonalities  among  them.  Using  those  reeurrent  elements,  a  model  will 
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be  developed  for  building  aequisition  me  trios.  The  literature  suggests  that  the  me  trios 
from  the  model  should  be  systematioally  developed  using  three  general  steps:  1)  oreate  a 
foundation  the  metrios  will  be  based  upon  (i.e.,  what  the  metrios  are  intended  to  foous  on 
and  be  built  from),  2)  identify  the  oritioal  elements  that  you  want  to  measure,  and  3) 
oreate  speoifio  metrios  for  those  oritioal  elements  in  order  to  improve  the  system’s 
performanoe.  Aooording  to  the  literature,  the  model’s  metrios  should  also  have  five 
attributes  oommonly  found  among  good  metrios:  relatedness  to  the  organization’s 
strategio  goals  and  mission,  simplioity,  meaningfulness  to  oustomers,  understandability, 
and  oost  effectiveness  (AFSC,  1991:2-1;  Antanitus,  2003:11;  Beamon,  1999;  Brown, 
1996:3-10;  Buchheim,  2000:311;  Evans  and  Lindsay,  2002:455,466;  Keebler  and  others, 
1999:118-121;  INCOSE,  1998:9;  Milliken,  2001;  Pinker  and  others,  1997:193;  Rummler 
and  Brae  he,  1995:138).  In  addition,  the  literature  indicates  that  it  is  preferred  for  metrics 
to  be  quantifiable,  but  that  sometimes  qualitative  metrics  more  adequately  discuss  system 
performance;  the  choice  between  qualitative  and  quantitative  depends  on  the  nature  of  the 
subject  matter  being  measured  (Beamon,  1999;  Locke,  1978:600).  To  better  understand 
the  applications  of  the  model  for  broad  use  with  any  large  acquisition  initiative,  a  case 
study  will  be  conducted  by  applying  the  model  to  the  2002  acquisition  Lightning  Bolt 
initiative  “Locus  on  results,  not  process.”  Chapter  three  will  discuss,  in  detail,  the 
methodology  to  be  used  in  this  study. 
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III.  Methodology 


The  goal  of  this  research  is  to  suggest  a  model  for  developing  performance 
measures  for  corporate  Air  Force  level  acquisition  based  initiatives.  Because  this 
research  is  exploratory,  it  will  focus  on  only  one  initiative;  the  2002  Lighting  Bolt 
initiative  “Focus  on  results,  not  process.”  To  do  this,  interviews  were  conducted  and  the 
interview  data  were  translated  into  metrics  for  the  initiative.  A  qualitative  analysis  of  the 
interview  data  was  also  done  using  a  protocol  based  on  those  of  Carter  and  Jennings 
(2002:145-179),  Creswell  (1997:142-146;  2003:196-215),  Isabella  (1990:7-41),  and 
Leedy  and  Ormrod  (2001:95,98,196).  A  variety  of  themes  and  patterns  emerged  from  the 
interview  data  through  the  analysis.  Subsequently,  the  level  of  success  the  Lightning 
Bolt  initiative  is  having  was  also  examined.  From  those  results,  metrics  are  suggested  for 
application  against  the  “Focus  on  results,  not  process”  initiative  and  for  broader, 
generalized  use  with  any  large  acquisition  based  initiatives.  A  description  of  the 
interview  process  is  provided  below. 

Methodological  Overview 

Based  on  the  nature  of  the  research  question,  a  qualitative  approach  was  used  to 
guide  the  research  project  based  on  the  procedures  outlined  by  Creswell  (2003:179-215). 
In  the  current  qualitative  research  effort,  data  was  collected  through  a  series  of  semi- 
structured  interviews  (Creswell,  2003:210-215).  After  the  interviews  were  transcribed, 
the  transcriptions  were  broken  down  into  statements  and  analyzed  for  common  themes. 
Specifically,  the  interviews  were  designed  to  generate  a  list  of  metrics  that  can  be  used  to 
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measure  the  Lightning  Bolt  of  interest  and  determine  the  extent  to  whieh  this  Lightning 
Bolt  is  appropriately  faeilitating  desired  ehanges. 

The  qualitative  study  findings  are  envisioned  to  be  reinforeed  by  future,  follow-on 
quantitative  researeh  that  should  support  and  further  validate  the  qualitative  findings, 
while  expanding  the  analysis  to  a  large  representative  sample.  As  part  of  the  future 
quantitative  researeh,  these  qualitative  data  will  be  translated  into  a  questionnaire  that  ean 
be  used  to  evaluate  the  list  of  metries  generated  and  to  gather  more  insight  into  the 
Lightning  Bolt’s  appropriateness  from  a  broader  audienee.  The  findings  from  both 
phases  should  be  integrated  as  part  of  future  researeh  where  the  quantitative  data  will  be 
used  to  reinforee  and  eonfirm  the  qualitative  results.  In  the  subsequent  seetions,  the 
nuanees  of  this  methodology  are  explained. 

Interview  Sample. 

In  2001,  Aequisition  Centers  of  Exeellenee  (ACE)  were  established  for  the  Air 
Eoree,  Air  Eoree  Materiel  Command,  Air  Eoree  Produet  Centers,  and  Air  Logistie 
Centers  to  lead  aequisition  reform  efforts  (New  Aequisition  Center  Provides  Warfighting 
Capabilities,  2001;  Eightning  Bolts,  2004).  Part  of  their  duties  is  to  oversee  the 
implementation  of  the  Eightning  Bolt  initiatives.  As  a  result,  the  ACE  offiees  have 
helped  system  program  offiee  (SPO)  leadership  understand  and  implement  the  initiatives. 
Therefore,  in  this  researeh,  members  at  the  ACE  offiees  and  various  system  program 
offiees  (SPOs)  within  the  Air  Eoree’s  Produet  Centers  and  Air  Eogisties  Centers  were 
interviewed.  To  further  broaden  the  researeh  sample,  individuals  holding  various 
aequisition  related  positions  within  Air  Combat  Command,  Air  Eoree  Spaee  Command, 
and  Air  Staff  were  also  interviewed.  Modeling  Carter  and  Jennings  (2002:150),  the 
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sample  interviewed  was  chosen  with  the  intent  of  getting  a  high  degree  of  variation 
among  managerial  levels,  in  order  to  get  a  higher  range  or  scope  of  data. 

The  ACE  personnel  were  asked  to  identify  interview  participants,  within  the 
SPOs,  who  have  experience  with  the  2002  Lightning  Bolt  initiative  of  interest.  The  letter 
requesting  ACE  office  assistance  in  identifying  interview  participants  is  shown  in 
Appendix  A.  The  letter  describes  the  target  interview  audience  as  consisting  of  40-50 
SPO  members  evenly  distributed  among  the  Product  and  Logistics  Centers,  holding 
various  levels  of  managerial  responsibility,  and  having  experience  using  the  “Locus  on 
results,  not  process”  Lightning  Bolt.  The  letter  also  asks  the  Secretariat  of  the  Air  Lorce 
Acquisition  Center  of  Excellence  (SAL/ACE)  office  to  utilize  subordinate  Center  ACEs 
to  identify  interview  subjects  within  this  target  audience  who  would  then  be  contacted  to 
participate  in  this  research. 

Of  the  fourteen  Center  ACEs  queried  for  assistance  by  the  SAE/ACE,  two 
provided  contact  information  for  interview  participants.  The  two  respondent  Center 
ACEs  were  from  separate  locations;  participants  from  Acquisition  Category  (ACAT)  one 
and  two  programs  were  identified  at  one  location  and  from  ACAT  three  programs  at  the 
other  location.  ACAT  describes  program  size  and  dollar  amount  and  ranges  from  one, 
being  the  largest  and  most  expensive  programs,  to  three,  being  the  smallest  and  least 
expensive.  Six  Center  ACEs  gave  negative  replies  (three  of  which  were  initially  non¬ 
respondent,  but  gave  negative  replies  when  asked  again)  and  cited  several  reasons  why: 
individuals  at  their  location  had  no  experience  with  the  initiative  of  interest;  they  sent  a 
message  out  to  SPOs  asking  for  participants  and  got  no  replies  back;  due  to  the  nature  of 
the  mission  at  their  location  (e.g.,  a  test  and  evaluation  organization)  they  did  not  use  the 
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initiative  of  interest;  they  never  reeeived  the  request  from  the  SAF/ACE  asking  for 
assistanee  with  the  researeh;  and  sinee  their  loeation  was  neither  a  Produet  nor  Logisties 
Center  (e.g.,  a  Test  Center),  they  thought  the  request  for  researeh  assistanee  did  not  apply 
to  them.  Some  Center  ACEs  eited  more  than  one  reason  for  their  negative  replies.  The 
six  remaining  Center  ACEs  were  eompletely  non-responsive  even  after  being  queried  a 
seeond  time. 

When  the  low  Center  ACE  response  rate  was  observed,  it  was  realized  that  a 
broader  interview  sample  was  needed  and  that  individuals  with  aequisition  experienee 
from  Air  Combat  Command,  Air  Eoree  Spaee  Command,  and  Air  Staff  should  be 
ineluded  among  interview  partieipants.  Additional  partieipants  were  identified  with  the 
assistanee  of  the  SAE/ACE,  through  interview  partieipants  reeommending  that  other 
speeifie  individuals  be  eontaeted  for  interviews  (also  referred  to  as  ‘snowball  sampling’), 
and  through  personal  eontaets  of  the  researeher.  A  total  of  twenty  five  partieipants  were 
identified  and  interviewed,  but  only  twenty  three  interviews  were  usable;  nineteen  verbal 
interviews  were  sueeessfully  transeribed,  two  verbal  interview  reeordings  were  inaudible 
and  subsequently  unusable,  two  interviews  were  reeorded  using  only  notes  taken  during 
the  interviews,  and  two  interviews  were  eondueted  via  email. 

Interview  Correspondenee. 

Potential  interview  partieipants  were  identified  and  then  eontaeted  to  determine 
their  willingness  to  partieipate.  Upon  their  agreement  to  assist  with  this  researeh, 
partieipants  were  sent  a  formal  invitation  letter  from  the  SAE/ACE  offiee.  This  letter, 
displayed  as  Appendix  B,  explains  that  the  researeh  is  designed  to  help  develop  metries 
of  sueeess  for  the  Eightning  Bolt  of  interest.  It  also  explains  that  any  data  gathered  from 
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the  interviews  will  remain  eonfidential  and  that  participants  will  not  be  identifiable.  In 
addition,  the  letter  contains  a  list  of  the  interview  questions.  The  interview  questions 
were  provided  prior  to  conducting  the  interviews  so  that  participants  could  prepare,  in 
hopes  of  making  the  interviews  more  efficient  and  effective. 

After  interview  candidates  were  identified  and  invited  to  participate,  each  was 
contacted  via  email  or  telephone  to  schedule  an  interview  time.  Prior  to  the  scheduled 
interviews,  each  participant  was  contacted  to  confirm  his  or  her  availability  for  the 
interview.  Then,  the  interviews  were  conducted  face-to-face,  over  the  telephone,  and 
using  email.  Notes  were  taken  during  each  verbal  interview,  and  interviews  were 
recorded  and  transcribed  (with  the  interviewee’s  permission)  whenever  possible. 

Appendix  C  is  a  global  appendix  of  the  additional  exchanges  that  took  place  with 
interview  subjects.  A  schedule  reminder  was  emailed  to  participants  to  reiterate  the  time 
and  date  of  each  interview.  The  reminder  also  acted  as  a  confirmation  of  participants’ 
availability  for  the  interviews.  Also  included  in  Appendix  C  is  a  copy  of  the  interview 
script  that  was  used.  The  script  addresses  the  intent  of  the  research  to  develop  metrics  for 
the  Lightning  Bolt  of  interest,  and  the  assurance  that  the  data  collected  during  the 
interviews  will  remain  confidential.  The  interview  questions  address  subjects’  views 
about:  a)  the  purpose  and  goals  of  the  “Focus  on  results,  not  process”  Lightning  Bolt 
initiative;  b)  metrics  to  be  used  to  measure  the  successful  implementation  of  the 
initiative;  c)  their  role  in  the  initiative’s  implementation;  and  d)  support  being  received  to 
implement  the  initiative. 
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Interview  Method. 


The  interviews  ineluded  open-ended  items,  allowing  participants  to  go  in  different 
directions.  However,  in  this  research,  a  semi-structured  interview  approach  was  used  in 
order  to  address  the  topics  of  interest  about  participants’  use  of  the  Lightning  Bolt,  within 
the  interview  time  constraints.  A  semi-structured  interview  enabled  the  research  to 
follow  the  standard  questions  in  Appendix  C  while  allowing  the  latitude  to  include  a  few 
tailored  questions  to  probe  or  clarify  a  participant’s  reasoning.  (Leedy  and  Ormrod, 
2001:196) 

Following  Carter  and  Jennings  (2002:152)  and  Leedy  and  Ormrod  (2001:98),  face 
validity  of  the  interview  questions  has  been  assessed  using  several  methods.  First,  the 
questions  were  reviewed  by  knowledgeable  academicians  and  acquisition  practitioners. 
Based  on  this  expert  review,  redundant  questions  were  removed;  a  few  questions  were 
reworded  to  prevent  leading  the  participants  and  to  make  the  questions  more  objective; 
and  a  few  questions  were  added  based  on  the  reviewers’  suggestions.  Second,  after  ten 
percent  of  the  interviews  were  conducted,  they  were  transcribed  and  reviewed  to 
determine  whether  themes  were  emerging  and  if  questions  were  clear  to  participants;  the 
interview  questions  were  refined  and  adjusted  accordingly.  Interviewing  is  a  dynamic 
process,  so  data  was  continually  analyzed  throughout  the  process  and  the  interview 
questions  adjusted  as  needed. 
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Analysis  Technique 

Data  Analysis. 

The  qualitative  data  collected  was  analyzed  using  Creswell’s  (1997:142-146)  data 
analysis  spiral.  Using  this  spiral,  data  was  reviewed  multiple  times  while  going  through 
the  following  steps: 

1.  Organization:  Organize  the  data  using  a  computer  database.  Break  large  portions 
of  text  into  smaller  units  (i.e.,  sentence  and  individual  words). 

2.  Perusal:  Peruse  the  whole  data  set  many  times  to  get  the  big  picture  of  what  it 
contains  as  a  whole.  Write  down  potential  interpretations  and  categories  while 
perusing. 

3.  Classification:  Identify  recurrent  themes  and  categories,  and  classify  or  group 
each  datum  accordingly.  In  this  step  patterns  should  begin  to  emerge. 

4.  Synthesis:  Integrate  and  summarize  the  data.  Develop  propositions  or 
hypotheses  that  describe  categorical  relationships.  Create  diagrams,  tables, 
matrices,  etc.  to  illustrate  proposed  relationships. 

Validation. 

Creswell  (2003:196)  suggests  eight  strategies  for  validating  the  accuracy  of 
findings,  three  of  which  applied  to  the  findings  of  this  research.  The  first  validation 
strategy  employed  is  to  clearly  state  the  biases  the  researcher  brings  to  the  study  and 
those  that  exist  due  to  the  nature  of  the  research  (Creswell,  2003:196).  The  following  are 
potential  biases.  First,  the  self-reporting  nature  of  interviews  makes  them  inherently 
biased.  Second,  the  interviews  only  look  at  a  snapshot  in  time  in  terms  of  the 
participants’  responses,  meaning  the  data  are  transient  in  nature,  so  interview  responses 
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may  be  different  at  different  points  in  time  (Leedy  and  Ormrod,  2001:95).  Third,  there 
eould  be  a  possible  sample  bias  beeause  partieipants  are  not  being  seleeted  randomly. 
However,  follow-on  researeh  eould  eounter  this  bias  by  randomly  seleeting  a  sample  that 
represents  the  population  of  Air  Foree  aequisition  praetitioners. 

Peer  debriefing  is  the  seeond  strategy  to  validate  the  findings  of  this  researeh. 
Experieneed  aequisition  praetitioners  who  are  familiar  with  this  area  of  researeh  reviewed 
and  asked  questions  about  the  qualitative  portions  of  the  researeh.  This  was  done  to  help 
ensure  the  accuraey  of  the  findings  and  to  make  the  explanation  of  this  researeh  elearer 
for  an  outside  audienee  (i.e.,  readers  other  than  the  researeher).  (Creswell,  2003:196) 

Following  Isabella  (1990:13)  and  utilizing  Creswell’s  (2003:196)  third  strategy, 
external  auditors  were  used  to  review  the  entire  researeh  projeet.  As  part  of  the 
qualitative  data  analysis,  reeurrent  themes  were  identified  and  interview  data  eategorized 
aeeordingly.  Non-aequisition  and  aequisition  professionals  who  were  new  to  the  researeh 
projeet  were  given  a  list  of  statements  from  the  interviews  and  a  list  of  the  themes  that 
emerged.  These  individuals  were  then  asked  to  eategorize  the  interview  statements  under 
the  themes  they  thought  were  appropriate  matehes.  The  purpose  of  this  was  to  see  if  the 
external  auditors  found  that  the  same  interview  statements  represent  the  themes  as 
intended.  The  independent  eategorization  provided  by  the  auditors  validates  the 
elassifieation  and  synthesis  of  the  data.  The  external  auditors  assessed  the  projeet  at  the 
eonelusion  of  the  researeh  proeess. 

The  qualitative  analysis  of  the  interview  data,  eombined  with  the  steps  to  ereate 
metrics  and  characteristics  of  good  metrics  that  were  revealed  through  the  literature. 
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enable  the  building  of  metries  for  the  initiative  of  interest.  The  following  seetions 
deseribe  what  was  gleaned  from  the  literature. 

Literature  Findings 

The  literature  reviewed  for  this  researeh  was  used  to  identify  a  set  of  three 
generalized  steps  for  ereating  metries  and  a  eomprehensive  list  of  desirable  qualities  for 
metries.  Table  2  and  Table  3  (previously  shown  in  ehapter  two)  illustrate  the  way  these 
steps  and  attributes  were  eaptured,  respeetively,  the  matriees  of  steps  and  attributes,  and 
the  authors  who  have  identified  the  various  steps  and  attributes. 

Consolidation  of  Steps  for  Creating  Metries. 

Methods  for  ereating  metries  suggested  by  multiple  authors  in  the  literature  were 
eompared  to  one  another  and  searehed  for  eornmon  elements.  Similarities  between  the 
methods  were  merged  into  one  generalized  three  step  proeess  for  ereating  metries.  The 
three  step  method  ineludes:  1)  deeide  what  is  to  be  measured,  2)  identify  the  eritieal 
aspeets  of  the  item  to  be  measured,  and  3)  ereate  speeifie  metries  for  eaeh  eritioal  aspeet 
in  order  to  improve  performanee.  The  metries  suggested  by  this  researeh  will  not  only  be 
developed  using  this  systematie  proeess,  they  will  embody  the  attributes  of  good  metries. 

Attribute  Identifieation. 

After  all  the  literature  was  reviewed,  the  list  of  attributes  identified  was  funneled 
down  into  a  eore  set  of  five  eornmon  attributes  to  eliminate  redundaney.  Attributes 
agreed  upon  by  four  or  more  authors  are  ineluded  in  the  eore  set.  Those  attributes 
inelude:  relatedness  to  the  organization’s  strategie  goals  and  mission,  simplieity, 
meaningfulness  to  eustomers,  understandability,  and  eost  effeetiveness  (AFSC,  1991:2-1; 
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Antanitus,  2003:11;  Beamon,  1999;  Brown,  1996:3-10;  Buchheim,  2000:311;  Evans  and 
Lindsay,  2002:455,466;  Keebler  and  others,  1999:118-121;  INCOSE,  1998:9;  Milliken, 
2001;  Pinker  and  others,  1997:193;  Rummler  and  Braehe,  1995:138). 

Core  Attributes. 

The  first  dimension  of  good  metrics  to  be  measured  is  that  they  relate  to  the 
strategic  goals  and  mission  of  the  organization  involved  (Antanitus,  2003:1 1;  Beamon, 
1999;  Buchheim,  2000:31 1;  Evans  and  Lindsay,  2002:455;  Milliken,  2001;  Pinker  and 
others,  1997:191).  This  relationship  is  important  so  the  organization  can  determine  if  it  is 
meeting  its  strategic  goals,  and  so  people  within  the  organization  will  focus  on  what  is 
being  measured,  thereby  steering  the  direction  of  the  organization  (Beamon,  1999). 

The  second  dimension  is  that  the  metric  must  be  meaningful  to  the  customer 
(AESC,  1991:2-1;  Brown,  1996:6;  Buchheim,  2000:311;  Pinker  and  others,  1997:193; 
Rummler  and  Braehe,  1995:138).  A  metric  is  meaningful  when  it  is  something  the 
customer  cares  about  (Buchheim,  2000:3 11).  And  it  must  present  data  that  enables  action 
to  be  taken  (AESC,  1991:1-1). 

The  third  dimension  of  good  metrics  is  simplicity  (AESC,  1991:2-1;  Buchheim, 
2000:31 1;  INCOSE,  1998:9;  Pinker  and  others,  1997:193).  Simplicity  means  that  the 
metric  must  be  as  simple  and  logical  as  possible,  so  it  will  be  easy  to  collect,  analyze,  and 
understand  (INCOSE,  1998:9). 

The  fourth  dimension  good  metrics  need  is  that  they  must  be  derivable  from 
economically  collectible  data  (AESC,  1991:2-1;  INCOSE,  1998:9;  Keebler  and  others, 
1999:1 19-121;  Pinker  and  others,  1997:193).  An  organization  must  decide  if  it  is  cost 
effective  to  collect  data  for  each  metric.  The  organization  has  to  determine  if  they  can  or 
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cannot  afford  to  collect  the  data,  and  if  the  data  offers  greater  value  than  it  costs 
(INCOSE,  1998:9). 

The  fifth  dimension  is  that  good  metries  must  be  understandable  (AFSC,  1991:2- 
1;  Buchheim,  2000:31 1;  Keebler  and  others,  1999:1 19;  Pinker  and  others,  1997:193). 
When  a  metrie  can  convey,  with  just  a  eursory  level  look,  how  it  was  derived  and  what 
exactly  it  is  measuring  then,  Keebler  and  others  (1999:1 19)  say,  it  is  easy  to  understand. 

Now  that  the  five  core  attributes  have  been  defined,  the  data  gathered  from  the 
interviews  can  be  analyzed  and  metries  for  the  initiative  of  interest  developed  by  using 
the  three  general  steps  for  ereating  metries  and  ensuring  that  the  ereated  metries  possess 
the  good  attributes. 

Summary 

A  qualitative  method  approaeh  has  been  used  to  conduet  this  researeh  projeet. 
Data  from  the  qualitative  analysis  was  applied  to  the  information  gathered  from  the 
literature  about  the  steps  involved  in  ereating  metries  and  the  eharacteristics  that  good 
metries  are  supposed  to  have,  in  order  to  develop  metrics  for  the  “Foeus  on  results,  not 
proeess”  Fightning  Bolt  initiative.  These  four  elements  together,  meaning  the  data  from 
interview  participants,  the  steps  to  creating  metries,  the  good  attributes,  and  the  initiative 
of  interest  used  as  a  case  study,  build  the  framework  for  developing  researeh  based 
metries  for  any  large  acquisition  based  program. 
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IV.  Data  Analysis 


The  focus  of  this  effort  is  to  create  a  framework  for  developing  measures  of 
success  for  corporate  level  Air  Force  acquisition  initiatives.  A  four  part  framework  has 
been  suggested.  Through  the  review  of  literature,  common  steps  for  creating  metrics 
were  established  and  recurrent  characteristics  of  good  metrics  were  identified.  Then 
interviews  were  conducted  with  acquisition  practitioners  who  have  experience  with  the 
initiative  of  interest.  Finally,  those  three  parts  will  be  applied  to  the  “Focus  on  results, 
not  process”  Lighting  Bolt  initiative  as  a  case  study  and  metrics  for  the  initiative  will  be 
suggested  as  a  result.  This  chapter  discusses  data  collected  during  the  interviews  and  the 
subsequent  analysis.  Multiple  patterns  and  themes  were  discovered  during  the  qualitative 
analysis  of  the  data.  Chapter  five  will  discuss  the  conclusions  drawn  from  the  analysis 
and  recommend  the  implementation  of  several  related  activities.  The  following  sections 
describe  the  data. 

Interview  Participants 

Interviews  were  conducted  with  Air  Force  officers  and  Air  Force  government 
civilians  from  Air  Combat  Command,  Air  Force  Materiel  Command,  Air  Force  Space 
Command,  and  the  Air  Staff  Individuals  ranged  in  rank  from  GS-12s  to  Senior 
Executive  Service  (SES)  members  and  General  Officers.  Participants  held  a  variety  of 
positions  from  System  Program  Office  (SPO)  level  program  managers,  contracting 
officers,  and  division  chiefs,  to  Program  Executive  Officers  (PEOs)  and  Center 
Commanders,  to  Secretary  of  the  Air  Eorce  staff  level  positions  (i.e..  Assistant  Secretary 
of  the  Air  Eorce  for  Acquisition  (SAE/AQ)  related  positions).  Their  time  in  federal 
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service  ranged  from  five  years  up  to  thirty  one  years.  The  high  degree  of  variety  in 
participants’  managerial  levels  and  areas  of  expertise  provided  a  high  range  or  scope  of 
data  (Carter  and  Jennings,  2002:150). 

For  purposes  of  maintaining  participant  confidentiality,  GM-15s,  GS-15s,  and 
Senior  Executive  Service  members  (excluding  those  within  the  Air  Staff),  and  System 
Program  Directors,  Program  Executive  Officers  (PEOs),  and  Center  Commanders  who 
participated  in  this  research  will  be  categorized  as  “middle  management.”  Interview 
participants  referred  to  this  grouping  of  people  as  middle  management,  to  Assistant 
Secretary  of  the  Air  Eorce  equivalent  positions  and  above  as  “senior  leadership,”  and  to 
Deputy  System  Program  Director  equivalent  positions  and  below  as  “SPO  level 
workers.”  Note  that  most  middle  managers,  and  some  workers,  are  actually  relatively 
senior,  experienced  personnel;  the  categories  are  essentially  a  self-classification  by  the 
participants  of  their  positions  relative  to  other  participants’  positions. 

Analysis  Overview 

Modeling  Isabella  (1990:13),  when  the  data  collection  was  completed,  interview 
participants’  responses  to  each  interview  question  were  systematically  and  carefully 
examined  to  identify  both  recurrent  and  unique  themes.  Every  interview  transcript  was 
reviewed  and  sections  of  the  interviews  were  excerpted  verbatim  and  typed  on  separate 
pieces  of  paper  to  illustrate  the  nucleus  of  each  individual’s  statements  (Isabella, 
1990:13).  After  excerpts  were  perused,  as  part  of  Creswell’s  (2003:142-146)  data 
analysis  spiral,  they  were  classified  into  recurrent  themes  and  categories.  Isabella 
(1990:13)  refers  to  this  as  coding  into  final  categories.  Roughly  seven  hundred  excerpts 
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were  reeorded.  Category  eoding  aeeuraey  was  ensured  using  external  auditors  (Creswell, 
2003:196;  Isabella,  1990:13).  Representative  examples  of  data  (i.e.,  interview  exeerpts) 
were  given  to  independent  reviewers,  or  external  auditors,  including  acquisition 
practitioners  and  non- acquisition  practitioners  new  to  the  research  project  (Creswell, 
2003:196;  Isabella,  1990:13).  The  reviewers  were  then  trained  in  the  rationale  used  for 
coding  excerpts  into  categories  (Creswell,  2003:196;  Isabella,  1990:13).  Due  to  the  large 
number  of  total  interview  excerpts,  reviewers  were  given  a  limited  number  of  excerpts  to 
code.  In  one  such  instance,  reviewers  were  asked  to  code  seventy  eight  excerpts  about 
what  metrics  should  be  used  for  the  initiative  of  interest  and  they  classified  fifty  six 
excerpts  into  the  same  categories  as  the  researcher,  giving  a  seventy  two  percent  level  of 
agreement  (Creswell,  2003:196;  Isabella,  1990:13).  Reviewers’  results  provided 
reasonable  verification  of  coding  procedure  accuracy  (Creswell,  2003:196;  Isabella, 

1990: 13).  The  patterns  and  themes  revealed  through  this  coding  are  described  below  in 
greater  detail  for  each  interview  question.  As  the  focus  of  this  research  is  developing 
measures  of  success  for  acquisition  initiatives,  the  themes  and  patterns  that  emerged  from 
the  interview  questions  directly  pertaining  to  metrics  for  the  initiative  of  interest  will  be 
addressed  first  and  the  findings  from  the  remaining  interview  questions  discussed 
thereafter. 

Suggested  Metrics  to  Measure  the  Success  of  “Focus  on  results,  not  process” 

Initiative  Implementation. 

Two  interview  questions  directly  addressed  metrics  for  the  initiative  of  interest. 
Interview  participants  were  asked  what  metrics  they  would  use  to  measure  the  results  the 
initiative  was  meant  to  bring  about,  and  then  later  in  the  interview  participants  were 
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asked  how  they  would  know  if  they  were  sueeeeding  at  implementing  the  initiative.  Out 
of  participants’  responses,  five  main  categories  of  metrics  were  recurrent:  schedule, 
customer  satisfaction,  cost,  performance,  and  credibility. 

The  theme  most  identified  by  participants  was  schedule.  Schedule,  also  called  an 
acquisition  program  baseline,  refers  to  the  lengths  of  time  a  program  has  set  to 
accomplish  various  tasks.  Based  on  participant  responses,  the  category  of  schedule  also 
includes  a  sub-category  of  cycle  time,  meaning  the  length  of  time  from  identifying  a  need 
for  something  until  it  is  delivered.  One  way  to  classify  cycle  time  is  by  whether  it  is 
oriented  around  capability  or  around  documentation.  Capability  based  cycle  time  refers 
to  the  amount  of  time  between  the  warfighter  stating  his  need  for  a  new  capability  (e.g., 
being  able  to  detect  enemy  troop  movements  within  buildings)  and  that  capability  is 
being  delivered  (e.g.,  an  infrared  sensor  is  installed  on  an  aircraft).  Documentation  based 
cycle  time  refers  to  acquisition  lead-time  or  the  time  it  takes  to  complete  a  document 
related  activity  (e.g.,  the  time  it  takes  for  a  contract  to  be  negotiated  until  the  legal 
document  is  signed  by  the  parties  and  processed  out).  This  theme  is  directly  related  to 
Sambur’s  (Senate  Armed  Services  Committee,  2002:1|1 1)  call  for  improved  speed;  the 
acquisition  community  has  to  deliver  things  when  they  say  they  are  going  to  deliver 
them.  Seventy  four  percent  of  participants  identified  schedule  as  a  metric  category  for 
the  initiative  of  interest. 

Customer  satisfaction  with  the  product,  process,  or  service  being  provided  was  the 
second  most  frequently  named  metric  category.  In  the  participants’  view,  customer 
satisfaction  also  encompasses  a  sub-category  of  expectation  management.  A  large  part  of 
how  satisfied  the  customer  is  depends  on  whether  they  received  what  they  were 
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expecting.  Sixty  one  percent  of  participants  named  customer  satisfaction  as  a  metric 
category. 

Cost  was  the  third  most  identified  theme.  This  theme  is  self  explanatory;  it  deals 
with  activities  related  to  money.  This  theme  occurred  among  thirty  nine  percent  of 
participants. 

Performance  was  the  fourth  most  recurring  theme  for  these  interview  questions. 

It  refers  to  how  well  or  how  poorly  a  product,  process,  or  program  is  performing 
compared  to  program  specifications  and  customer  expectations.  Both  customer 
satisfaction  and  performance  address  expectations;  of  the  two,  performance  is  the  more 
direct  comparison  against  expected  capability,  and  customer  satisfaction  addresses  a 
more  comprehensive  assessment  of  all  customer  expectations.  Twenty  six  percent  of 
participants  suggested  metrics  that  fit  into  this  category. 

Credibility  was  the  fifth  category  of  participant  interview  responses.  Credibility 
for  the  acquisition  workforce  would  mean  that  their  customers,  mainly  the  warfighters, 
would  believe  what  they  tell  them  and  find  them  trustworthy.  This  also  ties  in  with 
Sambur’s  (Senate  Armed  Services  Committee,  2002;1|1 1)  call  to  improve  credibility;  the 
acquisition  community  has  to  deliver  what  they  say  they  are  going  to  deliver.  However, 
the  occurrence  of  this  theme  among  only  seventeen  percent  of  participants  does  not  seem 
to  support  Sambur’s  push  for  improved  credibility;  this  indicates  that  not  many  people 
see  measuring  credibility  as  a  way  of  telling  if  this  initiative  is  succeeding.  In  contrast, 
the  frequent  recurrence  of  schedule  among  seventy  four  percent  of  participants  does  offer 
support  for  Sambur’s  call  for  speed. 
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The  remaining  interview  exeerpts  for  these  questions  that  did  not  seem  to  fit  into  a 
partieular  eategory  were  plaeed  in  a  miseellaneous  eategory.  Following  Isabella’s 
(1990:13-22)  example,  Table  4  displays  representative  exeerpts  from  responses  to  the 
interview  question  “What  metries  would  you  use  to  measure  those  results  or  outeomes?” 
and  illustrates  how  responses  flowed  aeross  each  of  the  five  categories  described  above. 
Table  5  displays  the  same  information  as  Table  4  for  to  the  second  specific  metrics 
related  question  of  “How  would  you  know  if  you  are  succeeding  at  implementing  the 
initiative?”. 

The  rest  of  the  chapter  discusses  themes  that  emerged  from  answers  to  the 
remaining  fifteen  interview  questions.  These  additional  questions  pertain  to  areas  of 
supplemental  interest  to  the  sponsor  of  this  research  project,  and  some  are  related  to  the 
development  of  generic  acquisition  initiatives.  The  conclusions  and  recommendations 
reached  from  the  data  analysis  will  be  discussed  later. 
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Schedule _ Customer  Satisfactioit _ Cost _ Per&rmaitce _ Credihili 

I  would  say  that  in  the  acquisition 
business  and  the  major  weapons 
system  acquisition  business  and 
even  if  we’re  talking  in  terms  of 


Table  4.  Interview  Exeerpt  Categorization  for  Suggested  Initiative  Metries 
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And  another  cntena  that  i  would  you  able  to  give  at  least  as  much 
use  IS  how  well  has  it  been  out  of  your  money  as  you  thought 

accepted  by  the  user. . .  You  you  were  going  to  get.  Is  your 

See  if  program  is  sticking  to  its  know,  managing  user  expectations  benefit  for  doing  this  thing  as  good 

Acquisition  Program  Baseline  is  a  large  part  of  that. _ as  you  thought  it  was  going  to  be. 


Table  5.  Interview  Excerpt  Categorization  for 
Measuring  Initiative  Implementation  Success 
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Meaning  of  2002  Lighting  Bolt  “Focus  on  results,  not  process”  Initiative. 

Interview  participants  were  asked  what  they  thought  the  initiative  of  interest 

meant.  Interview  participants  included  individuals  who  helped  to  draft  the  initiative,  one 

of  whom  stated  the  following  about  what  the  initiative  was  intended  to  mean: 

Too  many  people  within  the  acquisition  community  focus  on  completing 
processes  (reports,  assessments,  checklists,  etc).  The  Lighting  Bolt  aimed  to 
cause  people  to  look  at  the  result  intended  by  the  process  and  to  make  a  judgment 
of  whether  the  activity  planned  actually  furthers  the  opportunity  for  success. 
Success  isn’t  getting  through  the  process  -  its  delivering  a  needed  capability  to 
the  warfighter! 

A  variety  of  responses  were  provided  by  participants  and  then  grouped  by  the  themes  that 
emerged.  The  main  themes  that  emerged  are  listed  below  in  order  of  how  frequently  they 
occurred,  from  highest  to  lowest.  For  several  interview  questions,  some  excerpts  applied 
to  more  than  one  category  or  theme  (see,  for  example,  excerpts  within  the  categories  of 
Table  4  and  Table  5).  And  when  relevant  and  necessary  for  clarity,  more  than  one 
excerpt  was  selected  from  a  participant  in  order  to  capture  the  nucleus  of  their  response 
and  in  order  to  reflect  how  adamantly  they  responded  to  the  interview  question  (i.e.,  they 
stated  their  response  to  an  interview  question  multiple  times,  with  differing  explanatory 
nuances  in  each  response);  this  explains  why  excerpt  frequency  counts  are  higher  than  the 
number  of  participants  in  several  of  the  following  sections.  For  example,  the  first  theme 
that  arose  for  this  interview  question  was  from  ten  excerpts  shared  among  ten 
participants,  while  the  second  theme  came  out  of  nine  excerpts  from  eight  participants, 
and  the  other  themes  occurred  among  four  participants  or  less.  Appendix  D  summarizes 
descriptive  statistics  for  each  theme. 

•  Focus  on  the  end  customer  not  the  acquisition  process  itself;  support  the 
customer 
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•  Does  what  we  are  doing  make  sense  and  does  it  add  value?  If  not,  get  rid  of  it 
or  waive  it;  remove  the  unneeessary  steps 

•  Freeing  people  up 

•  Foeus  on  getting  the  produet  out;  effeets  based  or  outeome  based  aequisitions 

•  Want  results  not  just  proeess 

•  Clean-sheet  approaeh 

•  The  initiative  has  little  meaning  and  little  use 

•  Sets  the  stage  for  spiral  aequisition  and  evolutionary  aequisition 

•  Risk  management 

•  Proeess  must  serve  results 

•  Changing  people’s  mindset  to  look  at  what  they  ean  do  versus  what  they 
eannot 

Desired  Results  or  Outeomes  of  Initiative. 


Next,  partieipants  were  asked  what  they  thought  were  the  desired  outeomes  or 
results  that  the  initiative  was  trying  to  aeeomplish.  Multiple  themes  were  identified  from 
the  interview  data,  the  first  of  which  occurred  within  eleven  excerpts  among  nine 
respondents,  the  second  from  ten  excerpts  between  eight  respondents,  and  the  remainder 
from  five  excerpts  among  four  respondents  or  less.  The  following  are  the  themes  for  this 
question  in  order  of  frequency.  Included  in  Appendix  D  is  a  summary  of  descriptive 
statistics  for  each  theme. 

•  Support  the  Agile  Acquisition  strategy;  provide  capability  in  a  timely  way 
without  getting  bogged  down  in  the  processes 

•  Change  people’s  way  of  thinking;  be  creative,  innovative,  and  use  common 
sense 

•  Get  people  to  think  about  the  outcome  not  the  how 

•  Promises  made,  promises  kept 

•  Roadblocks  exist  to  accomplishing  initiative  outcomes  from  1)  middle 
management,  2)  SAF/AQ  staff  and  other  services,  and  3)  contracting 

•  Unchain  the  process  and  make  bureaucrats  look  at  the  big  picture 

•  Goals  are  unclear 

•  Challenge  decision  makers  when  necessary;  risk  management 

•  Freeing  up  the  workforce,  empowering  them 


48 


Appropriateness  of  Initiative  Goals. 

Participants  were  also  asked  if  they  thought  the  goals  of  the  initiative  were 
appropriate.  One  of  the  authors  of  the  initiative  described  the  goals,  or  outcomes,  of  the 
initiative  in  the  following  way: 

. . .  it's  from  Secretary  Roche,  General  Jumper,  Dr.  Sambur  and  at  the  time. 
General  Lyles,  Materiel  Command  Commander,  all  endorsed  the  speed  and 
credibility  as  the  two  primary  outcomes  that  they  wanted  from  this  —  from  the 
work  force  and  the  acquisition  system  and  it  was  this  Focus  on  Results,  Not 
Process  Lightning  Bolt  which  attempted  to  write  a  policy  on  how  we  do  business 
that  —  that  does  that.  The  outcome  is  the  result  itself,  not  the  process. 

Based  on  their  understanding  of  what  they  thought  the  goals  or  outcomes  of  the  initiative 

to  be,  twenty  of  the  twenty  three  participants  stated  that  they  felt  the  goals  were 

appropriate.  However,  when  asked  if  they  felt  using  this  initiative  was  the  most 

appropriate  way  to  accomplish  the  goals  that  it  was  meant  to  accomplish,  several 

participants  offered  various  criticisms  of  the  initiative.  A  few  of  those  criticisms  are 

listed  below. 

One  middle  management  participant  said: 

. . .  Well,  they  posted  these  lightning  bolts,  but  they  didn’t  give  me  background 
behind  them.  I  mean,  what  is  the  motive  for  this?  Sometimes  just  because  you 
put  something  out,  clear,  in  black  and  white  print,  unless  you  know  what  the 
under-pinning  motive  is  behind  it,  everyone  will  enact  upon  it  differently. 

Another  middle  management  participant  stated: 

...  At  the  time  of  the  Lightning  Bolts  -  as  long  as  Mrs.  Druyun  was  the  champion, 
you  could  roll  over  the  bureaucracy.  When  she  wasn’t,  there  wasn’t  any 
institutional  memory  to  show  why  or  how  you  could  have  waived  things. . . 

And  one  SPO  level  worker  remarked: 

. . .  It's  got  to  be  more  than  just  saying,  you  know,  this  is  a  Lightning  Bolt  and  we 
want  everybody  to  follow  it. . . .  It  has  to  be  a  top  down  mindset  change,  you 
know.  It  has  to  have  the  support  of  the  senior  leaders.  It  has  to  be  harped  on  over 
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and  over  and  over  again,  this  is  the  sorts  of  things  that  we're  trying  to  do. . .  And 
you  ean't  just  leave  it  for  the  —  the  aetual  day-to-day  employee  to  overeome  the 
inertia  of  things  the  way  they've  always  done  it...  And  there's  a  eouple  reasons  for 
that.  One,  it's  just  inertia.  Two,  it's  beeause  people  are  afraid  of  getting  squashed 
if  they  do  something  and  it  doesn't  work. . .  One  of  the  things  that  —  that  we  have 
to  be  real  eareful  about  is  —  is  punishing  failures  beeause  you  ean't  eome  up  with 
new  things  unless  you  try  things  and  fail  from  time  to  time. 

Most  Important  Aspeets  of  the  Initiative. 

The  interview  participants  were  then  asked  what  they  considered  to  be  the  most 
important  aspects  of  the  “Focus  on  results,  not  process”  initiative.  Four  main  themes 
were  identified  within  the  data  and  were  close  in  frequency  of  occurrence  among 
participants.  Organizational  culture  was  the  most  recurring  theme;  nine  excerpts  from  six 
participants  reflected  this  theme.  Participants  stated  that  an  entrepreneurial  mindset  was 
the  next  most  important  aspect  which  includes,  but  is  not  limited  to,  becoming  creative, 
not  being  risk  averse,  taking  bold  steps  to  challenge  the  status  quo,  and  thinking 
differently.  Seven  interview  excerpts  among  seven  participants  noted  this  aspect. 
Responsiveness  to  the  customer  was  the  next  most  frequently  seen  theme  with  six 
excerpts  among  five  participants.  Lastly,  five  excerpts  from  four  participants  shared  the 
theme  of  communication.  Appendix  D  includes  summary  descriptive  statistics  for  each 
theme. 

How  Participants  Heard  about  the  Initiative. 

Next  participants  were  asked  how  they  had  actually  heard  of  the  initiative.  Four 
participants  said  that  they  had  not  heard  of  the  “Focus  on  results,  not  process”  concept  as 
a  formal  initiative  until  they  were  contacted  about  this  research  project;  but,  based  on 
their  interview  responses  they  had  actually  already  been  carrying  out  the  intent  of  the 
initiative  within  their  jobs.  Those  participants  included  two  SPO  program  managers,  a 


50 


Deputy  SPO  Director,  and  an  Air  Staff  member.  Among  those  participants  who  had 
heard  of  the  initiative,  the  sources  from  which  they  learned  of  the  initiative  were  varied. 
The  most  frequent  source  of  introduction  to  the  initiative  was  through  participants’  chains 
of  command  and  normal  information  distribution  channels;  seven  excerpts  from  seven 
participants  shared  this  theme.  The  next  most  recurring  theme  was  direct  involvement 
with  Druyun,  the  originator  of  the  initiative.  Six  excerpts  from  five  participants  shared 
this  theme.  Three  excerpts  from  three  participants  noted  direct  contributions  to  writing 
the  initiative.  And  the  three  remaining  themes  observed  from  single  excerpts  among 
individual  participants  were  acquisition  reform  training,  Sambur’s  (Department  of  the  Air 
Force,  4  February  2003)  letter  to  the  acquisition  community  introducing  the  initiative  as 
part  of  the  new  push  for  improved  speed  and  credibility,  and  working  in  an  Acquisition 
Center  of  Excellence  (ACE)  office.  Appendix  D  summarizes  the  descriptive  statistics. 

Next  Step  in  “Eocusing  on  Results”. 

In  addition  to  being  asked  to  explain  how  they  had  heard  about  the  initiative, 
participants  were  asked  what  the  next  step  should  be  in  order  to  get  the  acquisition 
community  to  actually  implement  the  objectives  of  the  initiative  and  really  focus  on  the 
results.  There  were  as  many  responses  to  this  question  as  there  were  interview 
participants.  The  most  frequent  theme  within  the  interview  responses  for  this  question 
was  seen  within  six  excerpts  shared  among  four  participants.  The  second  most  frequent 
theme  came  from  six  excerpts  among  three  participants.  The  next  two  most  frequent 
themes  were  seen  in  three  excerpts  from  three  participants.  All  of  the  other  themes  were 
shared  by  only  two  participants  or  less.  Those  themes,  summarized  in  Appendix  D,  are 
as  follows: 
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•  Change  the  aequisition  workforce  culture 

•  Apply  the  initiative  to  the  processes  that  support  Evolutionary  Acquisition 

•  Training  and  education 

•  Expectation  management  with  the  warfighters  and  Air  Staff,  and  in  turn 
Congress 

•  Stop  smacking  people’s  hands  for  doing  acquisitions  differently  than  it  has 
always  been  done 

•  Corporate  buy-in  across  oversight  organizations  within  the  Air  Eorce  and  the 
Department  of  Defense 

•  Senior  Air  Eorce  leadership  buy-in 

•  Provide  examples  of  specific  success  stories 

•  Road  shows 

•  Be  specific  about  what  constitutes  “results” 

•  Move  people  from  oversight  to  execution  roles 

•  Take  people  out  of  the  approval  chain  who  are  not  in  the  decision  chain 

•  Bust  roadblocks 

•  Improve  filtering  down  of  initiatives;  initiatives  lose  a  lot  of  punch  by  the  time 
they  get  to  the  SPO  level  workers 

•  Release  a  new  set  of  initiatives  on  a  more  practical  and  specific  level 

•  Obtain  feedback  from  the  troops 

•  Took  at  how  we  make  acquisitions  work  in  the  future 

•  Include  demonstration  of  initiative  implementation  as  part  of  appraisals 

•  Send  out  messages  about  when  programs  and  people  failed,  but  were  still 
rewarded  for  trying  and  being  innovative 

•  Set  a  standard  or  cut-off  point  where  failing  programs  are  turned  off 

•  Get  people  willing  to  take  risks 

•  Obtain  buy-in  from  functional  and  operational  communities 

•  Eollow  through 

Organization  Implementation  of  the  Initiative. 

After  stating  what  they  thought  would  be  the  next  steps  to  take  in  order  to  get 
people  to  accomplish  the  goals  of  the  initiative,  participants  were  asked  how  their 
organizations  were  implementing  the  initiative.  The  themes  from  the  data  describing 
organizational  use  of  the  initiative  are  listed  below.  The  most  frequent  theme 
incorporates  seven  excerpts  from  five  participants  who  are  members  of  various 
Acquisition  Center  of  Excellence  (ACE)  offices;  the  second  most  frequent  theme  was 
from  seven  excerpts  among  three  participants;  and  the  third  most  frequently  occurring 


52 


theme  was  shared  by  four  excerpts  from  four  participants.  Most  of  the  themes  that 
emerged  were  only  common  among  one  or  two  participants.  See  Appendix  D  for  a 
summary  of  descriptive  statistics  for  each  theme. 

•  ACE  offices  assist  programs  to  challenge  burdensome  processes  and  try  to 
influence  people  to  use  the  philosophy  of  the  initiative 

•  Stress  full  participation  of  Integrated  Product  Teams  (IPTs)  and  ensure  IPTs 
include  the  warfighters,  contractors,  and  contracting  officers 

•  Rewrote  Air  Force  Instructions  and  Air  Force  Federal  Acquisition  Regulation 
Supplements  and  other  guiding  documents  to  free  up  people 

•  Negative  replies  about  not  using  the  initiative 

•  Opportunity  management;  being  proactive  versus  reactive,  being  creative,  and 
using  risk  management 

•  Challenged  SPO  members  to  use  common  sense,  and  then  if  it  is  not  the  law 
and  does  not  make  sense  to  break  it 

•  Ensured  Requests  for  Proposals  and  proposals  are  outcome  based 

•  Developed  and  implemented  training  on  using  the  initiative 

•  Applied  initiative  as  appropriate  to  SPO  programs 

•  Used  internal  teamwork  to  change  SPO  mindset  from  no  unless  you  can  prove 
yes,  to  yes  until  a  roadblock  is  found  that  says  it  cannot  be  done 

•  Empowered  SPO  program  leads,  lessened  emphasis  on  functional  leads; 
removed  non  value-added  parts  of  the  chain  of  command 

Most  participants  indicated  they  were  implementing  the  initiative  in  some  way, 
but  several  said  they  were  not.  Most  notably,  several  middle  management  participants 
said  they  were  not  actively  implementing  the  initiative  because  from  their  perspectives 
Air  Force  Materiel  Command  is  now  more  process  oriented  than  product  oriented.  They 
are  even  being  sent  to  lecture  series  and  workshops  by  Michael  Hammer  on  how  to 
specifically  use  and  reengineer  processes  with  the  idea,  as  some  participants  noted,  that 
by  so  doing  they  will  later  be  able  to  focus  on  the  results.  Hammer  is  a  New  York  Times 
bestselling  author,  credited  with  creating  the  concepts  of  reengineering  and  process 
enterprise  (Hammer,  2001  :i).  Another  participant  stated  that  they  would  not  remove  non 
value-added  acquisition  processes  because  they  would  not  challenge  the  Federal 
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Acquisition  Regulation  without  a  lawyer.  And  one  other  partieipant  said  they  were 
having  diffieulty  implementing  the  initiative. 

Partieipants’  Roles  in  the  Development  or  Implementation  of  the  Initiative. 

Not  only  were  partieipants  asked  how  their  organizations  were  implementing  the 
initiative,  each  participant  was  asked  what  his  or  her  specifie  role  was  in  either  the 
development  or  implementation  of  the  initiative.  The  most  reeurring  theme  was  that 
partieipants  acted  as  enablers  for  their  teams.  Those  enabling  aetivities  included  but  were 
not  limited  to  the  following:  ehallenging  their  teams  to  use  the  initiative;  running 
interferenee  for  their  teams  when  their  attempts  to  implement  the  initiative  met 
resistanee;  massaging  relationships  (i.e.,  developing  and  maintaining  a  good  working 
relationship)  with  people  involved  with  the  aequisition  proeesses  to  whieh  they  were 
trying  to  apply  the  initiative  and  making  sure  things  were  running  smoothly.  The  theme 
of  being  an  enabler  was  formed  from  fourteen  exeerpts  among  eight  partieipants.  All 
other  themes  eame  from  three  exeerpts  from  three  partieipants  or  less.  The  themes,  also 
summarized  in  Appendix  D,  are  listed  below. 

•  Being  an  enabler  for  your  team 

•  Developer  or  author  of  the  initiative 

•  Endorser  and  advoeate  of  the  initiative 

•  Had  no  role  in  the  development  of  the  initiative 

•  Provide  adviee  to  senior  leadership  on  ways  to  implement  the  initiative 

•  Helped  rewrite  polieies  as  a  result  of  using  the  initiative 

•  Provided  training  for  SPOs  on  how  to  use  the  initiative 

•  Managed  customer  expeetations 

•  Arehiteet  for  implementation  of  the  initiative  within  a  SPO 

Support  for  Organizational  Implementation  of  Initiative. 

After  partieipants  were  asked  about  their  roles  in  the  development  and 
implementation  of  the  initiative,  they  were  asked  several  questions  about  the  level  of 
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support  they  are  receiving  in  their  attempts  to  implement  the  initiative.  The  first  of  these 
questions  asked  what  kind  of  support  participants’  organizations  were  getting  as  they  try 
to  use  the  initiative.  The  most  recurring  theme  that  was  seen  in  responses  to  this  question 
came  from  eighteen  excerpts  given  by  thirteen  participants.  The  other  themes  were 
expressed  in  three  excerpts  among  three  participants  or  less.  The  themes  about  types  of 
organizational  support  are  listed  below  in  descending  order  of  frequency  of  occurrence. 

•  Top  down  support 

•  ACE  help  in  planning  for  program  events 

•  Contractor  support 

•  Initiative  training;  risk  management  training  and  Discovery  Map  training 

•  Being  left  alone  and  trusted  to  go  implement  the  initiative  is  the  best  support 

•  Internal  support 

•  Congressional  language  can  be  a  big  supporter 

However,  several  negative  themes  about  the  level  of  support  organizations  were  receiving 
arose  from  the  responses  of  seven  participants. 

•  No  support  is  being  given 

•  The  bureaucracy  is  fighting  implementation  of  the  initiative 

•  Senior  leaders  empowered  the  workforce  to  go  out  and  implement  the 
initiative,  but  they  are  not  preaching  it  enough  themselves;  need  strong, 
consistent  advocacy 

•  Headquarters  puts  the  initiatives  out  but  does  not  have  to  live  with  them 
Appendix  D  summarizes  both  the  positive  and  negative  themes  raised  in  responses  to  this 
question. 

Support  for  Individual  Implementation  of  Initiative. 

The  next  support  related  question  dealt  with  whether  or  not  they  felt  they  were 
getting  the  support  they  needed  to  implement  the  initiative.  Over  sixty  five  percent  of  the 
participants  said  they  were  receiving  the  support  they  needed  to  implement  the  initiative 
from  those  within  their  chain  of  command  and  from  those  areas  within  their  control. 
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However,  thirty  pereent  of  partieipants  said  they  were  not  getting  the  support  they  needed 
from  those  who  are  outside  of  their  ehain  of  eommand  but  ean  still  influenee  their  ability 
to  implement  the  initiative.  Five  pereent  of  participants  were  undecided.  Participants 
noted  that  they  were  not  getting  support  from  Headquarters  Air  Force  (HQ  USAF),  Air 
Staff,  or  the  Office  of  the  Secretary  of  Defense.  They  cited  the  need  to  have  a  constant 
push  from  senior  Air  Force  leaders  in  order  to  implement  the  initiative.  For  example,  one 
participant  said: 

. . .  where  I  don't  have  the  support  is  the  stuff  that  the  senior  people  need  to  do, 
they  aren't  doing  enough.  I  mean,  they  —  they  just  —  they  —  they're  distracted  by 
day  to  day  tactical  problems  of  running  the  Air  Force  to  the  point  where  —  where 
if  they  have  some  time,  they'll  do  transformation  stuff,  but  they  —  I  —  I  just  get  the 
sense  that  they  don't  get  that  without  them  involved  on  a  —  on  a  almost  daily 
basis,  the  —  the  —  the  forces  against  change  just  are  so  strong  that  they'll  —  they 
figure  they  can  wait  it  out. 

While  another  participant  noted  what  senior  leaders  could  ideally  do  to  provide  support: 

...  in  my  ideal  world.  Chief  of  Staff  of  the  Air  Force  would  —  would,  you  know, 
come  out  and  once  a  month,  he'd  have  some  speech  on  why  performance  based 
contracting  or  results  based  acquisition  is  critical  to  the  success  of  the  Air  Force. 
And  every  time  an  issue  came  up  that  dealt  with  contractors  or  acquisition,  we'd 
have  a  consistent  message  on  that. 

The  need  for  consistent,  repeatedly  vocal  support  from  senior  Air  Force  leaders  and  the 
need  for  buy-in  from  people  and  processes  outside  of  the  immediate  Air  Force  chain  of 
command  that  can  still  heavily  impact  people’s  successful  implementation  of  the 
initiative  continue  to  be  recurring  themes. 

Organizational  Support  for  Individual  Implementation  of  Initiative. 

Participants  were  also  asked  how  their  organizations  were  supporting  them  in 
their  attempts  to  put  the  initiative  into  action.  The  theme  of  support  and  encouragement 
being  provided  by  leadership  within  participants’  direct  chains  of  command  was  noted 
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among  twelve  excerpts  from  eleven  out  of  twenty  three  participants.  Three  other 
participants,  including  two  middle  management  members,  said  they  received  support 
from  their  organizations  by  being  trusted  to  do  the  job  and  being  left  alone  to  do  it.  Three 
excerpts  from  one  participant  called  out  strong  support  from  the  ACE  offices  as  an 
avenue  of  organizational  support.  Another  participant  said  they  were  getting  support 
from  their  organization  by  virtue  of  having  no  kick-back  from  SPO  members  which 
indicated  that  the  SPO  members  have  accepted  the  challenge  for  their  organization  to 
implement  the  initiative.  The  final  theme  that  arose  out  of  excerpts  for  this  question  was 
from  a  participant  who  twice  stated  that  they  were  not  seeing  leaders  at  the  Senior 
Executive  Service  (SES)  and  General  Officer  level  engage  enough  in  the  drive  to  use  this 
initiative;  the  participant  considers  leaders’  involvement  to  be  one  of  the  most  important 
tools  they  need  to  do  their  job.  See  Appendix  D  for  a  summary  of  descriptive  statistics 
for  each  theme. 

Initiative  Implementation  Success  Stories. 

After  participants  were  asked  about  the  level  of  support  they  were  receiving  in 
their  attempts  to  utilize  the  initiative,  they  were  asked  if  they  had  heard  of  any  success 
stories  or  failures  at  using  the  initiative.  Ten  of  the  twenty  three  participants  said  they 
could  cite  no  specific  examples  of  success  stories,  but  eleven  other  participants  did 
provide  examples  of  what  they  considered  to  be  successes.  The  success  stories  were 
grouped  into  two  categories:  process  level  successes  and  program  level  successes. 

Process  level  successes  are  examples  of  ways  the  initiative  of  interest  has  been 
used  to  remove  non  value-added  processes  and  which  can  be  repeated  within  program 
offices  across  the  entire  Air  Eorce.  One  such  example,  as  noted  by  one  participant,  was 
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the  addition  of  a  source  selection  plan  into  a  System  Acquisition  Management  Plan  in 
order  to  get  approval  for  both  at  the  same  time.  Another  participant  discussed  the 
incorporation  of  a  Price  Competition  Memo  (PCM)  in  a  Proposal  Analysis  Report  (PAR) 
which  reduced  time  because  now  the  same  pricing  structure  can  be  used  for  both  the  PAR 
and  PCM. 

In  another  process  level  success  story,  a  participant  was  trying  to  purchase 
desktop  computers  and  related  software  to  be  used  in  an  Air  Force  office  and  was  told 
through  their  contracting  approval  chain  that  the  base  lawyers  said  a  Mission  Need 
Statement  (MNS)  and  Operational  Requirements  Document  (ORD)  would  be  needed  due 
to  the  dollar  threshold  of  the  computer  purchase.  The  participant’s  office  challenged  the 
requirement  for  the  MNS  and  ORD  because  it  did  not  make  sense  since  the  purchase  was 
just  for  office  computers.  Their  challenge  was  successful  and  the  requirement  for  the 
MNS  and  ORD  was  done  away  with.  That  challenge  process  took  about  three  days  and 
saved  them  six  months  of  work  had  they  been  required  to  do  the  MNS  and  ORD. 

Another  example  of  putting  the  initiative  to  work  involved  a  reduction  in  training 
approval  time.  An  office  was  taking  thirty  to  forty  days  to  get  training  classes  for  their 
acquisition  workforce.  The  process  was  examined  and  it  was  discovered  that  training 
notifications  were  being  held  up  significantly  by  base  training  officers  who  were  not 
directly  involved  in  approving  the  training,  but  who  merely  wanted  feedback  to  track  who 
was  going  to  training.  As  a  result,  the  process  was  changed  by  removing  the  non  value- 
added  steps  and  base  training  officers  now  get  feedback  on  people  who  attend  training  on 
the  back  end  (i.e.,  after  the  approvals  have  been  made  they  receive  computerized  tracking 
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information  at  the  same  time  people  are  notified  of  their  training  approval),  and  training 
notifieations  are  eomputerized  and  now  have  a  twenty  four  hour  turnaround. 

Additionally,  a  partieipant  stated  that  the  delegation  of  eontraet  approval  authority 
down  to  various  base  level  personnel,  so  that  people  now  rarely  have  to  go  through  the 
headquarters  for  eontraeting  (i.e.,  SAF/AQ  and  SAF/AQC),  is  also  an  initiative 
implementation  sueeess  story.  Non  value-added  steps  were  removed  and  people  not 
essential  to  the  approval  proeess  were  removed  from  the  deeision  ehain  of  eommand. 

In  addition  to  proeess  level  sueeess  stories,  several  weapon  system  programs  were 
reeognized  by  participants  as  examples  of  how  the  initiative  can  be  utilized  successfully. 
One  such  program  is  the  Crystal  Modification  Program.  The  organization  running  that 
program  was  able  to  go  and  influence  the  Army  and  Navy  to  combine  functionalities  of 
boxes  where  it  made  sense  to  do  so  and  reduce  the  footprint,  and,  in  turn,  reduce  the 
development  costs  and  infrastructure  costs.  The  focus  remained  on  the  product  and  non 
value-added  steps  were  eliminated.  According  to  participants,  reducing  the  footprint  of 
the  cryptographic  systems  required  across  the  Department  of  Defense  (DoD)  is  a  success 
story  that  saves  the  DoD  money  and  gets  capabilities  met  more  effectively  and  more 
efficiently. 

Several  other  programs  were  also  identified  as  success  stories  because  of  how 
they  kept  their  focus  on  the  results  being  delivered  to  the  warfighters  and  how  they  did 
not  get  bogged  down  in  the  acquisition  process  itself  Programs  like  Global  Hawk,  Micro 
Impulse  Radar,  Patient  Support  Pallet,  and  the  weaponization  of  Predator  are  additional 
examples  that  participants  considered  success  stories  of  how  the  initiative  of  interest  can 
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be  used.  However,  one  final  program  was  considered  to  be  a  story  both  of  success  and  of 
failure,  namely  the  ChemSentry  Chemical  Detector  program. 

One  hundred  ChemSentry  Chemical  Detectors  were  purchased  as  Commercial- 
Off-the-Shelf  equipment  in  January  2003.  According  to  participants,  this  buy  was  made 
to  support  warfighters  in  Operation  Iraqi  Freedom  and  was  called  the  “best  of  agile 
acquisition”  by  the  Aeronautical  Systems  Center  Acquisition  Executive.  Many  of  the 
traditional  contracting  processes  and  testing  procedures  were  streamlined  to  get  the 
product  out  to  the  field  quickly.  The  chemical  detectors  put  out  in  the  field  were  not 
completely  tested,  but  they  were  fielded  very  quickly  to  meet  an  urgent  need  and  they  did 
give  some  level  of  protection  (better  than  essentially  no  protection  at  all).  Several 
interview  participants  considered  this  to  be  a  success  story  in  the  sense  that  the  program 
was  able  to  get  an  increased  level  of  protection  in  the  hands  of  the  warfighters  out  in  the 
field.  Prior  to  fielding  the  detectors,  troops  were  using  chickens  to  detect  chemical  and 
biological  warfare;  during  air  attacks  chickens  were  put  outside  to  see  if  they  died  or  not 
to  tell  troops  if  a  deadly  agent  was  present.  This  was  one  more  example  of  a  program 
level  success  story  of  initiative  implementation.  In  contrast,  the  follow-on  ChemSentry 
buy  was  considered  to  be  a  failure  at  using  the  initiative. 

Initiative  Implementation  Failures. 

Forty  three  percent  of  interview  participants  indicated  that  they  had  not  heard  of 
any  specific  examples  of  failure  at  using  the  initiative,  though  several  did  identify 
program  level  and  process  level  failures.  As  noted  above,  however,  some  participants 
said  the  ChemSentry  program  failed  in  its  attempts  to  use  the  initiative  during  the  follow- 
on  buy  of  one  hundred  additional  chemical  detectors  when  agencies  outside  of  the  Air 
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Force  became  involved.  When  the  Army,  who  managed  the  chemical  detectors 
previously  used  by  the  Air  Force,  became  aware  of  the  new  ChemSentry  purchase  and 
fielding  they  performed  an  independent  assessment  of  available  chemical  detectors  and 
recommended  that  the  old  type  of  chemical  detector  be  used  to  fill  the  Air  Force 
requirements  instead.  Due  to  the  Army’s  involvement,  and  subsequently  that  of  the 
Office  of  the  Secretary  of  Defense  (OSD),  additional  testing  of  the  detectors  was 
required,  the  program  was  forced  to  go  through  the  burdensome  and  lengthy  steps  of  the 
traditional  acquisitions  process,  and  leadership  became  risk  averse  and  did  not  support 
the  use  of  the  initiative.  The  participants  said  the  program  is  now  likely  to  be 
discontinued. 

In  addition  to  the  ChemSentry  Chemical  Detector  program,  two  other  programs 
were  said  to  have  failed  at  implementing  the  initiative.  Two  middle  management 
members  interviewed  noted  that  the  F/A-22  aircraft  program  was  an  example  of  a  failed 
attempt  to  use  the  initiative.  And  another  participant  described  the  increase  in  manpower 
requirements  by  the  Program  Executive  Officer  (PEO)  offices,  recently  relocated  to  the 
Air  Eorce  Materiel  Command  Product  Centers,  over  their  previous  manpower  levels  used 
while  located  at  the  Pentagon  as  a  failure  at  implementing  the  initiative  of  interest. 
Eailures  at  using  the  initiative  were  also  discussed  at  the  process  level. 

Examples  of  process  level  failures  at  initiative  utilization  were  noted  by  two 
participants.  One  participant  discussed  how  the  Air  Eorce’s  process  for  reprogramming 
funds,  which  allows  money  to  be  used  for  programs  other  than  what  it  was  originally 
slated  for,  does  not  enact  the  initiative.  The  impression  among  the  Air  Eorce  workforce 
is  that  Congress  is  why  it  takes  too  long  to  approve  reprogramming.  The  participant’s 
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office  checked  and  found  out  that  Congress  only  takes  thirty  of  the  hundred  and 
fifty-eight  day  cycle  to  reprogram  funding;  the  rest  is  taken  up  by  the  Air  Force.  So  if 
money  has  to  be  reprogrammed  above  certain  approval  thresholds,  it  takes  an  average  of 
a  hundred  and  thirty  days  just  to  process  the  request  through  the  Pentagon. 

The  other  process  level  failure  example  involved  an  attempt  to  do  a  zero  baseline 
of  all  work  in  a  SPO  at  Electronic  Systems  Center.  This  exercise  was  originally  designed 
to  challenge  the  value-added  contribution  of  every  activity  that  the  program  office  was 
doing  by  forcing  each  activity  and  report  to  justify  its  contribution.  SPO  personnel  were 
not  interested  in  doing  it.  This  was  clearly  a  failure  at  implementing  the  initiative.  After 
participants  were  asked  about  successes  and  failures  at  using  the  initiative  of  interest, 
they  were  asked  questions  about  acquisition  reform  initiatives  in  general. 

Being  Successful  at  Implementing  Any  Acquisition  Reform  Initiative. 

The  last  two  interview  questions  were  applicable  to  acquisition  reform  initiatives 
on  a  broader  scale.  The  first  of  these  two  questions  asked  participants  what  they  think  it 
takes  to  be  successful  at  utilizing  any  acquisition  reform  initiative.  A  range  of  themes 
emerged  from  their  responses.  The  most  frequently  occurring  theme  was  seen  among 
eleven  interview  excerpts  from  seven  participants.  The  second  most  frequent  theme 
came  from  seven  excerpts  among  seven  participants.  The  third  most  frequent  theme  was 
common  among  five  excerpts  from  five  participants.  And  the  fourth  most  common 
theme  was  from  six  excerpts  among  four  participants.  The  remaining  themes  arose  from 
four  excerpts  among  three  participants  or  less.  All  of  the  themes  about  what  it  takes  to 
make  any  acquisition  reform  initiative  successful  are  summarized  in  Appendix  D,  and 
listed  below  in  order  of  decreasing  frequency  of  occurrence. 
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•  Consistent  message  from  the  top 

•  Senior  leadership  buy-in;  support  and  advocacy  for  initiative  from  senior 
leaders 

•  Be  very  specific  in  what  the  initiative  says  and  in  what  is  expected  of  those 
who  use  it 

•  Behavior  of  leaders  has  to  reinforce  philosophy  behind  the  initiative 

•  It  takes  time  to  successfully  implement  an  initiative 

•  Non-acquisition  perspective;  initiative  should  be  written  for  a  broad  audience 
and  by  individuals  with  more  than  just  an  acquisition  background 

•  Equip  people  to  use  the  initiative;  teach  people  about  the  initiative,  train  them 
to  use  it,  provide  them  with  the  resources  to  implement  it 

•  Buy-in  from  middle  management  (i.e.,  GM/GS-15s,  Senior  Executive  Service 
members.  System  Program  Directors,  Center  Commanders,  and  PEG  level 
leadership) 

•  Marketing;  show  people  in  the  acquisition  trenches  how  the  initiative  will 
make  their  jobs  better,  more  effective  and  efficient,  and  how  it  will  help  the 
warfighters 

•  Buy-in  from  people  in  the  acquisition  trenches  and  their  desire  to  succeed 

•  More  than  just  support;  teams  have  to  be  pushed  to  change 

•  Teamwork 

•  Hold  people  accountable 

•  Eocus  on  changing  the  acquisition  culture 

•  Downplay  buzzwords  like  “acquisition  reform;”  those  words  are  overused  and 
people  outside  of  Product  Centers  think  they  do  not  apply  to  them 

•  Ingenuity  because  one  size  initiative  does  not  fit  all  programs 

•  Trust  of  leadership 

•  Eeadership  from  program  managers  not  functional  leads 

•  Ereedom  to  use  common  sense 

•  Empowerment 

•  Reality  based  acquisitions  management 

•  Enthusiasm 

•  Buy-in  from  operational  leadership 

•  Communication 

Important  Elements  of  an  Acquisition  Reform  Initiative. 

Easily,  after  interview  participants  were  asked  about  what  they  considered  to  be 
the  keys  to  successful  initiative  implementation,  they  were  asked  what  was  important  to 
them  in  any  acquisition  reform  initiative.  Many  of  the  themes  that  emerged  from  the  data 
mirrored  the  characteristics  of  good  metrics  found  within  literature.  The  most  recurring 


63 


theme  was  from  six  interview  exeerpts  among  five  partieipants.  The  next  three  most 
reeurring  themes  were  eaeh  eommon  among  four  exeerpts  from  four  partieipants.  The 
other  themes  produeed  eame  from  five  exeerpts  from  three  partieipants  or  less.  Eaeh 
theme  about  what  is  important  in  an  initiative  is  summarized  in  Appendix  D,  and  listed 
below  in  order  of  deereasing  frequeney  of  oeeurrence. 

•  Foeus  on  the  mission  of  the  Air  Force  and  getting  something  to  the 
warfighters 

•  Be  beneficial  to  the  acquisition  grunts  and  the  end  users 

•  Makes  sense 

•  Follow  through;  see  it  through  to  the  end 

•  Do  not  make  initiatives  just  so  they  are  easy  to  measure,  they  should  be  useful 

•  Knowing  the  motive  behind  the  initiative;  seeing  how  it  fits  into  the  big 
picture 

•  Expectations  management 

•  Improves  performance 

•  Clearly  defined;  it  is  communicated  well  with  examples 

•  Buy-in  for  it  from  all  levels 

•  Feadership  and  advocacy  for  the  initiative 

•  Frees  people  up  to  innovate 

•  Understandable 

•  Provides  specific  plan 

•  Cannot  be  restrictive 

•  Culture  changing 

•  Does  not  lose  what  is  good  about  current  efforts 

•  Attacks  other  than  on  the  margin  (i.e.,  it  attacks  funding  stability, 
requirements  stability,  expectations  management) 


Summary 

After  the  interview  data  were  carefully  examined  using  methods  modeled  after 
Creswell  (2003:142-146,196)  and  Isabella  (1990:7-41),  excerpts  from  participants’ 
responses  that  represented  the  core  of  their  answers  to  each  question  were  grouped  by  the 
themes  that  emerged.  Appendix  D  visually  summarizes  the  details  about  each  theme  that 
this  chapter  describes;  it  captures  all  of  the  recurrent  themes  that  emerged  from  the  data 
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during  the  qualitative  analysis,  the  total  number  of  excerpts  that  represented  each 
corresponding  theme,  the  total  number  of  interview  participants  who  provided  the 
excerpts,  and  the  percentage  of  participants  who  discussed  each  theme.  Those  themes 
revealed  participants’  opinions  about  the  kinds  of  metrics  they  would  use  to  measure  the 
“Focus  on  results,  not  process”  initiative’s  success,  the  meaning  and  goals  of  the 
initiative,  the  next  step  in  achieving  the  initiative’s  goals,  how  they  heard  about  it,  the 
kind  of  support  initiative  implementation  is  receiving,  successes  and  failures  at  initiative 
utilization,  and  how  to  make  generic  initiatives  successful.  Now  the  data  analysis  will  be 
combined  with  the  steps  to  create  metrics  and  attributes  of  good  metrics  from  the 
literature  and  applied  to  the  initiative  of  interest  to  draw  conclusions  and  make 
recommendations  for  further  action. 
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V.  Conclusions  and  Recommendations 


This  chapter  presents  eonclusions  drawn  from  the  data  analysis  and  makes 
implementable  reeommendations  to  help  the  sponsor  of  this  researeh  and  the  aequisition 
eommunity  more  effectively  utilize  the  initiative  of  interest.  This  researeh  has  attempted 
to  help  senior  Air  Foree  leaders  know  how  well  the  “Foeus  on  results,  not  proeess” 
initiative  is  working  and  build  a  framework  for  developing  measures  of  sueeess  for 
eorporate  level  Air  Foree  aequisition  initiatives.  That  metries  framework  was  built  by 
reviewing  literature  and  distilling  the  steps  involved  in  ereating  metries  down  into  three 
universal  steps  and  the  attributes  of  good  metries  into  a  eore  set  of  attributes,  by 
performing  a  qualitative  data  analysis  on  interview  data  gathered  from  aequisition 
praetitioners,  and  by  then  applying  those  elements  to  the  initiative  of  interest  as  a  ease 
study  and  reeommending  metries  for  the  initiative  as  a  result.  The  following  sections 
discuss  the  eonelusions  and  reeommendations  reaehed  through  this  researeh,  and  deseribe 
the  steps  involved  in  the  follow-on  quantitative  phase  of  the  researeh. 

Conclusions 

Five  eonelusions  were  drawn  from  the  interview  data.  A  breakdown  in 
eommunieation  about  what  the  initiative  meant  oeeurred  throughout  the  aequisition 
workforee;  no  eonsistent  definition  for  the  initiative  was  found  among  partieipants  exeept 
among  the  partieipants  who  helped  author  the  initiative  and  those  who  work  in  ACE 
offiees.  Next,  it  was  determined  that  diseonneets  exist  between  the  middle  management 
level  and  the  other  management  levels  on  several  fronts.  Middle  management  shared  a 
unanimous  view  on  what  the  initiative  was  intended  to  aeeomplish,  but  that  view  differed 
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from  the  view  that  senior  leaders  and  SPO  level  workers  shared.  Middle  management 
also  did  not  appear  to  see  the  eonneetion  between  the  initiative  and  the  Agile  Aequisition 
eoneept  that  both  senior  leaders  and  SPO  level  workers  did.  As  diseussed  in  detail  later, 
the  relationship  between  middle  management’s  diseonneets  with  the  other  management 
levels  may  be  due  in  part  to  some  bias  towards  the  initiative  originator.  Another 
eonelusion  is  that  partieipants  pereeive  that  administrative  hurdles  to  implementing  the 
initiative  are  being  put  up  by  organizations  and  ageneies  outside  of  partieipants’  ehains  of 
eommand.  It  was  also  eoneluded  that  there  are  differing  pereeptions  about  to  whom  the 
initiative  applies.  Lastly,  it  was  noted  that  the  use  of  the  initiative  may  not  be  the  best 
way  to  bring  about  the  desired  ehanges.  The  following  seetions  provide  a  more  in  depth 
deseription  of  the  eonelusions  reaehed  through  this  researeh. 

Breakdown  in  Communieation  of  Initiative  Definition. 

The  first  eonelusion  reaehed  through  the  data  analysis  is  that  there  was  a 
breakdown  in  eommunieation  of  what  the  “Foeus  on  results,  not  proeess”  initiative 
meant.  No  eonsistent  definition  for  the  initiative  was  presented  by  the  partieipants  exeept 
by  those  partieipants  who  helped  write  the  initiative  and  those  who  work  in  ACE  offiees 
helping  other  organizations  implement  the  initiative  (New  Aequisition  Center  Provides 
Warfighting  Capabilities,  2001;  Lightning  Bolts,  2004).  Many  people  took  the  initiative 
to  mean  that  if  the  aequisition  proeess  in  question  was  not  a  law,  then  break  it.  But, 
aeeording  authors  of  the  initiative,  what  was  intended  was  for  people  to  look  at  the  non 
value-added  proeesses  and  ehallenge  the  ones  that  do  not  make  sense,  and  make  a  ease 
for  why  a  proeess  should  not  be  used  and  get  a  waiver  for  it.  The  proeesses  were  not 
meant  to  be  ignored,  but  neither  were  they  meant  to  be  stumbling  bloeks  along  the  way  to 
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providing  a  capability  to  the  end  customer.  One  partieipant  who  works  elosely  with  the 
implementation  of  this  initiative  through  their  work  with  the  Aequisition  Center  of 
Excellenee  (ACE)  offices  summed  it  up  well  when  she  said: 

. .  they  [the  2002  Eightning  Bolt  initiatives]  were  deployed  but  there  wasn’t  a 
fanfare,  a  marketing  eampaign  -  there  wasn’t  enough  communication  with  the 
field  on  what  was  really  meant  by  it.  And  so,  what  happened  was,  I  think  people 
made  their  own  interpretations  as  what  was  meant  by  it  and  so  one  of  the 
consequenees  was  that  some  people  said,  hey,  diseipline  goes  out,  all  I  have  to  say 
is  I  want  to  do  this  in  the  spirit  of  transformation  the  spirit  of  vague  acquisition  I 
should  be  able  to  do  it,  right?  Well,  that  isn’t  neeessarily  true.  You  still  have  to 
provide  reasons  why  you  want  to  do  something  and  that  make  sense  eonsistent 
with  the  regulations  and  laws  that  we  have.  You  usually  ean  waive  things  but  you 
ean’t  just  waive  them  without  having  a  reason  to  waive  them.  You  still  have  to 
make  your  case  and  you  don’t  have  to  have  a  stack  of  papers  this  high  to  make  a 
case  but  you  still  have  to  make  case.  So  probably  you  may  want  to  skip  that  step, 
it’s  like,  I  just  want  to  do  it  so  I  should  be  allowed  to  do  it,  right?  Well,  no  you 
should  make  a  case  for  it.” 

Disconneets  Between  Management  Eevels. 

A  seeond  conelusion,  related  to  the  first,  is  that  there  is  a  diseonneet  between  what 
both  senior  Air  Eoree  leaders  and  SPO  level  workers  think  the  initiative  was  meant  to 
aeeomplish  and  what  the  middle  management  level  leadership  thinks.  The  people  at  the 
senior  leader  and  the  SPO  worker  ends  of  the  management  chain  seem  to  have  a  clearer 
understanding  of  what  the  Eightning  Bolt  was  supposed  to  mean  and  think  that  getting  rid 
of  the  non  value-added  proeesses  is  a  good  thing.  Eor  example,  senior  leaders  provided 
guidanee  on  how  to  apply  the  initiative  and  on  what  the  initiative  means  in  the  form  of 
policy  letters,  directives,  and  briefings  (Department  of  the  Air  Eoree,  4  Eebruary  2003; 
Department  of  the  Air  Eoree,  10  July  2003;  Senate  Armed  Serviees  Committee,  2002). 

In  addition,  SPO  level  workers  provided  numerous  examples  of  success  stories  at 
implementing  the  initiative  within  their  program  offiees,  showing  a  elear  understanding 
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of  the  initiative  and  aetive  use  of  it.  However,  middle  management  thinks  the  initiative  is 
a  “dead  horse,”  as  several  middle  management  partieipants  ealled  it,  and  they  think  that 
the  Air  Foree  is  not  even  really  pursuing  the  “Foeus  on  results,  not  proeess”  philosophy. 
Many  of  them  eited  the  Hammer  training  that  they  are  being  required  to  attend  whieh 
tells  them  that  proeess  is  everything  and  that  the  results  will  follow,  and  so  this  initiative 
is  eounter  to  the  training  they  are  reeeiving.  The  majority  of  middle  management 
partieipants  interviewed  shared  this  view,  though  that  is  not  to  say  that  they  all  thought 
the  approaeh  they  pereeived  the  Air  Foree  to  be  taking  (i.e.,  proeess  foeused  versus 
results  foeused)  was  the  right  one. 

The  disconneet  between  management  levels  may  be  partially  due  to  some  bias  the 
middle  management  appears  to  have  towards  the  originator  of  the  Lightning  Bolt 
initiatives.  This  is  visible  in  some  of  the  partieipants’  eomments.  For  instanee  one 
middle  management  member  said  “this  initiative  really  didn't  do  anything  to  me,  but  to  be 
honest  with  ya',  I  thought  it  was  a  parroting  of  something  that  Mrs.  Druyun  heard  out  of 
the  Chiefs  mouth...”  Another  middle  management  eommented,  “I'll  tell  you,  you  know 
the  tension  between,  and  tension  is  a  very  kind  term,  between  Dr.  Sambur  and  Darlene 
Druyun,  you  know,  I  —  I  ean't  imagine  there  being  any  way  that  he  would  fully  embraee, 
you  know,  the  things  that  you  left  behind. . .  Darlene  is  gone,  Darlene's  policies  are 
gone.”  And  a  third  middle  management  member  who  was  interviewed  said  “my  honest 
opinion  is  I'm  not  sure  it  [the  initiative]  means  much  of  anything,  to  tell  you  the  truth,  I 
mean,  when  Mrs.  Druyun  came  up  with  the  last  set  of  Lightning  Bolts,  she  was  just 
lookin'  —  I  mean,  it's  more  of  a  —  of  a  —  it's  more  of  a  buzz  word  to  me  than  anything, 
you  know,  ‘Focus  on  Results,  Not  Process’.” 
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In  addition,  middle  management  does  not  seem  to  see  any  eonnection  between  the 
Lightning  Bolt  initiative  and  the  Agile  Aequisition  philosophy.  One  middle  management 
partieipant  put  it  this  way: 

“But,  you  know,  Agile  Aequisition  is  good.  Get  the  bureaueraey  out  of  the  way, 
that's  the  —  that  starts  the  —  you  know,  the  policy  stream  of  Dr.  Sambur.  Better 
systems  engineering,  incentivizing  systems  engineering,  expectations 
management,  realistic  cost  estimates.  All  those  things  are  —  you  know,  are  now 
the  objectives  of,  you  know,  whatever  we  want  to  call  it,  reformation, 
transformation,  re-engineering. . .  What  you  don't  hear,  you  know,  focus  on 
results,  not  process.  That  —  that's  still  a  good  concept,  but  it's  not  what  people  are 
thinking  or  saying.” 

The  initiative  of  interest  was  meant  to  serve  as  one  of  the  tools  to  help  accomplish  Agile 
Acquisition  (Department  of  the  Air  Force,  4  February  2003).  The  initiative  and  Agile 
Acquisition  have  the  same  goals  of  improving  the  speed  and  credibility  of  the  acquisition 
community,  and  people  are  expected  to  tradeoff  non-critical  program  elements  (i.e.,  get 
rid  of  non  value-added  processes)  as  part  of  Agile  Acquisition  which  is  also  what  the 
initiative  was  meant  to  drive  (Department  of  the  Air  Force,  4  February  2003). 

Administrative  Obstacles  to  Initiative  Implementation  from  Outside  Agencies. 
Another  conclusion  is  that  several  administrative  obstacles  to  successfully 
implementing  the  initiative  exist.  Participants  clearly  stated  that  they  are  getting  the 
support  they  need  from  their  immediate  bosses,  but  that  there  are  obstacles  from  outside 
organizations  and  agencies;  for  example,  added  oversight  from  Congress,  having  to  work 
with  the  Department  of  Defense  and  other  services,  and  having  to  get  approvals  from 
people  outside  of  their  decision  chain  of  command.  Thirty  percent  of  participants  stated 
that  those  outside  their  chain  of  command,  who  can  still  influence  their  level  of  success  at 
implementing  the  initiative,  are  not  providing  the  support  participants  need.  Numerous 
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participants  also  included  the  Office  of  the  Secretary  Of  Defense  (OSD)  among  their 
biggest  perceived  roadblocks.  For  example,  one  former  PEO  said  they  agreed 
“absolutely,  one  hundred  percent”  with  other  participants  that  OSD  is  one  of  the  main 
places  they  start  hitting  roadblocks.  The  ChemSentry  program  follow-on  buy  of 
ehemical  testers,  discussed  previously  in  chapter  four,  was  cited  as  an  example  of  OSD’s 
counterproductive  involvements. 

Differing  Perceptions  About  Who  is  to  Implement  the  Initiative. 

In  addition  to  there  being  roadblocks  to  implementing  the  initiative,  people 
outside  of  the  acquisition  community  do  not  think  the  initiative  applies  to  them. 
According  to  some  participants,  the  dubbing  of  the  initiative  as  an  acquisition  reform 
initiative  led  people  within  the  requirements  arena  (e.g..  Air  Combat  Command),  people 
in  the  testing  community,  and  those  in  the  logistics  and  weapon  system  sustainment 
eommunity  to  think  the  initiative  was  only  geared  towards  weapon  system  acquisition 
offices.  The  negative  responses  from  many  of  the  people  approached  to  participate  in 
this  research  cited  the  point  that  they  did  not  think  the  initiative  applied  to  them  as  the 
reason  they  could  not  help  with  this  research;  Test  and  Evaluation  Centers,  Air  Eogistics 
Centers,  and  an  Air  Eorce  Space  Command  System  Program  Office  (SPO)  did  not 
participate  in  this  research  for  that  reason. 

Initiative  May  Not  Be  Most  Effective  Way  to  Bring  About  Desired  Changes. 

The  last  conclusion  made  from  the  data  analysis  is  that  using  an  initiative  like  this 
may  not  be  the  most  effective  way  to  aceomplish  the  desired  behavioral  change.  Just 
sending  out  an  initiative  and  leaving  it  up  to  the  workers  in  the  acquisition  trenches  to 
figure  out  how  to  apply  it  will  not  cut  it.  The  need  for  leadership  and  people’s  fears  of 
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change  and  failure  need  to  be  addressed.  Partieipants,  ineluding  the  middle  management 
personnel,  eonsistently  said  that  senior  leaders  need  to  be  more  engaged  and  regularly 
voeal  about  the  importanee  of  things  like  the  initiative  for  it  to  be  suecessful.  After 
eonclusions  were  reaehed,  reeommendations  were  developed  using  both  the  data  analysis 
and  literature  review  findings. 

Implementable  Recommendations 

Several  implementable  reeommendations  are  presented  in  the  following  seetions. 
The  first  reeommendation  was  developed  by  applying  the  framework  for  developing 
metries  that  this  researeh  has  produeed  to  the  “Foeus  on  results,  not  proeess”  initiative. 

As  previously  diseussed,  five  categories  of  metrics  are  suggested  for  the  initiative, 
speeifieally  sehedule,  eustomer  satisfaetion,  eost,  eredibility,  and  performanee.  The  first 
reeommendation  of  this  researeh  will  only  address  the  metric  category  of  customer 
satisfaetion  beeause  the  eategories  of  sehedule,  eost,  eredibility,  and  performanee  have 
been  previously  addressed  by  other  Air  Foree  ageneies  and  because  cost,  schedule,  and 
performanee  metrics  are  already  broadly  used  aeross  the  Air  Foree.  The 
reeommendations  of  this  researeh  attempt  to  focus  on  areas  that  will  be  of  the  most 
interest  and  utility  to  the  researeh  sponsor  and  to  the  Air  Foree. 

Customer  Satisfaction  Metries  to  Measure  Initiative  Sueeess. 

This  researeh  reeommends  the  use  of  eustomer  satisfaetion  metries  to  measure  the 
sueeess  of  the  “Focus  on  results,  not  proeess”  initiative.  Customer  satisfaction  is  the  key 
to  organizational  sueeess  (Gibson  and  others,  2003:238).  No  matter  how  preeisely  a 
sehedule  is  maintained,  how  mueh  eost  savings  are  realized,  how  eredible  the  end 
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customer  thinks  the  aequisition  eommunity  is,  or  what  exeeptional  performanee  a  weapon 
system  or  proeess  has,  if  the  customer  is  not  satisfied  with  the  result,  the  aequisition 
eommunity  has  failed.  Customer  satisfaetion  is  deseribed  as  the  extent  to  whieh  a 
proeess  or  produet  meets  a  eustomer’s  expeetations  (Kotler  and  Armstrong,  2001:9; 
Naumann  and  Jaekson,  1999:71;  Zeithaml  and  Bitner,  2006:75).  The  dimensions  of 
eustomer  satisfaetion  ean  be  applied  aeross  the  other  eategories  of  metries  (e.g.,  sehedule, 
eredibility,  and  performanee)  to  suggest  metries  for  the  initiative.  It  should  be  noted  that 
multiple  eustomer  satisfaetion  metries  eould  apply  and  one  size  does  not  fit  all  (INCOSE, 
1998:9);  the  following  are  examples  that  apply,  but  future  researeh  eould  provide 
additional  eustomer  satisfaetion  metries.  A  eustomer  satisfaetion  metrie  that  applies  to 
sehedule  is  timeliness  (Ellis  and  Curtis,  1995;  Hayes,  1992:8).  Customer  satisfaetion 
metries  for  performanee  are  reliability  and  pereeived  quality  (Ellis  and  Curtis,  1995; 
Naumann  and  Jaekson,  1999:72).  And  a  eustomer  satisfaetion  metrie  for  the  area  of 
eredibility  is  responsiveness  (Ellis  and  Curtis,  1995;  Hayes,  1992:8;  Naumann  and 
Jaekson,  1999:72).  A  eustomer  satisfaetion  metrie  for  eost  is  not  suggested  beeause, 
aeeording  to  Hammer  (2001 : 103),  it  tells  very  little  if  anything  about  the  business.  The 
visual  depletion  of  how  the  dimensions  of  eustomer  satisfaetion  ean  be  applied  aeross  the 
metrie  eategories  of  sehedule,  eost,  performanee,  and  eredibility  to  produee  generally 
applieable  eustomer  satisfaetion  metries  for  the  initiative  is  displayed  below  as  Table  6. 
The  list  of  metries  is  general  sinee  the  attributes  of  eaeh  dimension  are  very  produet 
speeifie,  meaning  the  metries  should  be  tailored  for  a  better  fit  depending  upon  what 
produet  or  proeess  they  are  applied  to  (Naumann  and  Jaekson,  1999:73).  One  size  metrie 
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does  not  fit  all  (INCOSE,  1998:9).  The  development  of  more  speeific  program  and 
proeess  metries  eould  be  explored  as  part  of  future  research. 


Table  6.  Generally  Applicable  Customer  Satisfaction  Metrics 


Dimensions  of  Customer  Satisfaction  Metrics 

Schedule 

Timeliness 

Performance 

Reliability  and  perceived  quality 

Credibility 

Responsiveness 

In  addition  to  suggesting  metrics  for  the  “Focus  on  results,  not  process”  initiative, 
this  research  makes  several  other  recommendations  to  improve  the  implementation  of  the 
initiative  and  future  initiatives.  These  additional  recommendations  are  discussed  in 
greater  detail  below.  Following  those  recommendations  are  suggestions  for  future 
research  and  a  summary  of  this  research. 

Innovation  Training  for  Acquisition  Workforce. 

The  acquisition  workforce  needs  to  continue  to  be  recruited  and  trained  to  think 
outside  the  box  and  to  not  always  follow  the  cookbook  recipe.  The  workforce  has 
traditionally  been  trained  in  how  to  use  the  Federal  Acquisition  Regulation  (FAR),  the 
Department  of  Defense  5000  Series,  source  selection  guides,  and  other  guides  and 
instructions  about  what  processes  and  procedures  to  follow,  but  if  the  acquisition 
community  is  now  expected  to  be  innovative,  not  have  the  business  as  usual  mindset, 
think  for  themselves,  be  creative,  and  change  how  acquisitions  are  conducted,  then  they 
have  to  be  trained  in  how  to  do  that.  There  is  a  need  for  this  kind  of  training  among  the 
workforce.  This  is  evidenced  by  one  middle  management  participant  who  was  speaking 
to  acquisition  practitioners  on  Fightning  Bolt  initiatives  and  a  woman  in  the  audience 
said,  “I  used  to  be  able  to  sit  down  at  my  desk  and  open  my  cookbook  and  follow  the 
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recipe  and  I'd  get  done  with  the  product.  You  took  my  recipe  book  away  from  me  and  I 
don't  know  what  to  do.”  The  objective  now  is  not  to  have  people  open  the  book  and 
follow  it  step  by  step,  but  instead  to  think  for  themselves  and  do  what  is  smart.  But  they 
have  to  be  trained  how  to  do  that.  One  avenue  to  train  the  acquisition  community  is  to 
train  people  in  modem  business  philosophies  and  tactics,  like  strategic  purchasing,  the 
entrepreneurial  mindset,  and  organizational  management,  versus  the  standard  government 
process  and  procedure  oriented  training. 

More  specifically,  several  contracting  division  chiefs  noted  that  contracting 
personnel  should  be  specifically  targeted  for  training  because  they  are  often  seen  as 
roadblocks.  The  division  chiefs  also  stated  that  many  contracting  people  are  very 
regimented  in  how  they  do  business  and  if  it  is  not  specifically  authorized  in  the  Federal 
Acquisition  Regulation  they  are  not  willing  to  do  it.  In  addition,  many  civilians  entering 
the  government  sector  today  are  taught  little  about  the  military  in  their  educations,  and  it 
is  unlikely  that  they  will  put  the  emphasis  on  the  warfighter  in  their  work  if  they  do  not 
have  an  understanding  of  what  the  warfighter  does  or  goes  through  on  the  battlefield. 
Participants  suggested  that  one  way  to  sensitize  government  civilians,  specifically  those 
in  the  contracting  career  field,  to  the  warfighters’  needs  is  to  include  them  in  Professional 
Military  Education  (PME)  classes.  This  is  being  done  to  some  extent  already,  but  it 
should  be  expanded.  Participants  also  distinctly  identified  the  need  for  personnel  at  the 
Defense  Einance  and  Accounting  Service  and  the  Defense  Contract  Management  Agency 
to  receive  PME. 
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Create  Functional  Area  Guiding  Principles. 

The  next  recommendation  is  that  all  functional  areas  (e.g.,  engineering,  finance, 
program  management)  within  the  Assistant  Secretary  of  the  Air  Force  for  Acquisition 
(SAF/AQ)  office  create  a  set  of  guiding  principles  for  the  people  within  their  functional 
areas  in  the  acquisition  community  that  would  spell  out  what  is  expected  of  those  people 
as  they  participate  in  this  transformation  movement.  The  guiding  principles  created  by 
SAF/AQC  for  contracting  personnel  could  act  as  the  template  for  the  other  functional 
areas.  There  is  a  need  for  an  expectations  guide;  participants  from  the  ACE  offices  noted 
that  they  get  numerous  calls  from  people  asking  what  is  expected  of  them  in  this  new 
acquisition  environment  and  from  people  asking  how  to  do  acquisition  without  the 
cookbook.  Principles  like  the  ones  SAF/AQC  has  included  in  part  one  of  the  Air  Force 
Federal  Acquisition  Regulation  (FAR)  Supplement  (Department  of  the  Air  Force,  1  May 
2003)  give  people  an  idea  of  what  they  are  expected  to  do,  how  they  are  expected  to  act, 
and  what  they  are  expected  to  accomplish. 

Make  Waiver  Process  Easier. 

The  next  recommendation  is  to  make  the  waiver  process  less  burdensome  and 
faster.  The  initiative  of  interest  is  designed  to  enable  acquisition  practitioners  to 
challenge  the  things  that  do  not  make  sense;  in  turn,  the  process  used  to  make  these 
challenges  needs  to  be  addressed.  It  takes  so  much  effort  and  time  to  get  a  waiver  for 
regulations.  Air  Eorce  Instructions,  and  other  required  processes  that  you  might  as  well 
have  just  done  the  process  or  followed  the  regulation  or  instruction  that  you  were  trying 
to  change.  Many  participants  said  the  current  method  is  just  too  much  trouble.  This  view 
is  supported  by  one  of  the  General  Officers  interviewed  who  stated  the  following. 
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“So  what's  important  for  me  in  an  initiative  is  that  it  be  something  I  ean  do  and  it 
would  be  value  added  and  it  would  aetually  out  my  work.  In  other  words,  don't 
tell  me. . .  I  ean  have  an  exoeption  to  somethin'  if  I  have  to  go  ask  for  —  you  know, 
I  have  to  go  sell  this  exoeption  to  every  layer  of  bureauoraoy  I've  gotta  work  with. 
It  just  doesn't  —  it's  —  if  s  just  here,  let  me  do  it  the  regular  way.” 

Clarify  Future  Initiatives. 

It  is  also  reoommended  that  the  next  set  of  initiatives  to  be  sent  out  not  use 
buzzwords  like  aoquisition  or  aoquisition  reform;  rather,  they  should  be  very  speoifio,  and 
metrios  and  a  oommander’s  intent  statement  should  be  released  along  with  the  initiatives. 
One  middle  management  partioipant  noted  that  when  the  term  aoquisition  reform 
initiative  is  used  “the  rest  of  the  Air  Foroe  thinks  it  only  applies  to  the  aoquisition  world.” 

Future  initiatives  should  also  be  well  defined  and  not  general  statements. 
Numerous  interview  partioipants  noted  that  the  most  reoent  series  of  Lightning  Bolt 
initiatives  greatly  differed  from  the  previous  initiative  releases  in  that  it  was  vague  and 
too  general.  Speoifioity  leaves  less  room  for  ambiguity  and  misinterpretation.  Assigning 
speoifio  goals  or  tasks,  whioh  in  this  ease  would  be  a  speoifio  initiative,  has  been  proven 
to  improve  performanoe  (Latham  and  Looke,  1979:68-80;  Latham  and  Yukl,  1975:824- 
843;  Looke,  1968:157;  Taylor,  2001:64).  Speoifio  initiatives,  in  turn,  will  aid  in  the 
oreation  of  speoifio  metrios. 

It  is  also  reoommended  that  measures  of  suooess  for  the  next  initiatives  be 
established  before  the  initiatives  are  released.  This  will  help  olarify  what  the  initiatives 
are  trying  to  aooomplish  and  what  ohanges  they  are  trying  to  bring  about,  and  help 
determine  when  the  initiative  has  been  aooomplished. 

In  addition,  a  oommander’s  intent  statement  should  aooompany  the  initiatives.  A 
oommander’s  intent  statement  will  give  those  expeoted  to  implement  the  initiatives  some 
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insight  into  the  thinking  behind  them,  and  will  allow  those  who  implement  them  the 
freedom  to  use  their  initiative  and  judgment  in  a  manner  that  is  eonsistent  with  the  aims 
of  higher  eommanders  (Department  of  the  Air  Foree,  10  July  2003:2). 

Create  Database  of  Initiative  Implementation  Sueeess  and  Failure  Stories. 

The  next  reeommendation  is  that  the  Seeretariat  of  the  Air  Foree  Aequisition 
Center  of  Exeellenee  (SAF/ACE)  offiee  eompile  a  database  that  eontains  examples  of 
sueeessful  and  failed  attempts  to  use  the  “Eoeus  on  results,  not  proeess”  initiative. 
Multiple  interview  partieipants  expressed  an  interest  in  seeing  sueh  a  database  ereated.  It 
would  serve  as  a  way  to  make  the  initiative  of  interest,  whieh  is  more  general  and 
eoneeptual  than  previous  Eightning  Bolt  initiatives,  more  praetieal  and  applieable  for 
aequisition  practitioners.  Several  participants  suggested  that  it  could  help  dispel  fear  and 
apprehension  about  trying  something  new  and  failing  by  providing  examples  of  programs 
that  failed  at  using  the  initiatives,  but  were  still  rewarded  for  being  innovative  and  for 
trying.  The  success  and  failure  examples  presented  in  this  research  can  serve  as  the 
starting  point  for  the  database. 

It  is  suggested  that  a  link  to  the  database  be  posted  on  the  SAE/ACE  internet 
homepage.  And  the  inclusion  of  a  feature  story  in  Agile  Acquisition:  The  Air  Force 
Acquisition  Newsletter  on  one  of  the  successes  or  failures  should  be  considered,  with  the 
monthly  stories  alternating  between  failure  and  success. 

ACE  Office  Road  Shows. 

If  they  are  not  already  doing  so,  it  is  also  recommended  that  the  SAE/ACE  office 
coordinate  with  the  Center  ACE  offices  to  present  road  shows  specifically  focused  on  the 
initiatives  they  are  overseeing,  like  the  initiative  this  research  is  focusing  on,  at  bases 
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throughout  the  Air  Force.  Road  shows  could  be  the  avenue  by  which  the  ACE  offices 
clearly  convey  the  intent  behind  the  initiatives  and  explain  what  they  were  meant  to 
accomplish.  Road  shows  would  be  a  display  of  leadership’s  support  for  and  advocacy  of 
the  initiatives.  They  could  be  the  path  to  train  and  educate  people  about  the  initiatives 
and  obtain  feedback  from  the  troops  on  initiative  implementation.  Additionally,  they 
could  be  a  time  to  share  stories  of  successful  initiative  utilization  and  stories  where 
people  or  programs  were  not  successful. 

Consistent  Statement  of  Initiative  Support  from  Senior  Air  Force  Leaders. 

Lastly,  it  is  recommended  that  the  Chief  of  Staff  of  the  Air  Force  be  asked  to 
include  a  statement  of  support  for  continued  acquisition  reform,  stressing  the  importance 
of  compliance  with  acquisition  initiatives  in  his  monthly  Chief’s  Sight  Picture  newsletter. 
And  it  is  recommended  that  the  Secretary  of  the  Air  Force  be  asked  to  make  a  similar 
statement  in  his  Secretary ’s  Vector  newsletter.  Such  statements  would  address  the 
prevailing  and  pervasive  request  from  interview  participants  and  others  for  consistent, 
continual,  vocal  support  for  the  initiative  of  interest  from  the  Chief  and  other  senior  Air 
Force  leaders.  The  Chief  and  the  Secretary’s  continued  and  open  support  would  enable 
those  expected  to  implement  the  initiatives  to  really  challenge  the  party-line  way  of  doing 
acquisitions,  become  innovative,  and  change  the  acquisition  process. 

This  research  has  made  several  recommendations  to  better  measure  the  success  of 
the  initiative’s  performance,  and  to  improve  its  implementation  and  that  of  future 
initiatives.  Eight  recommendations  are  made  in  total.  The  first  is  to  use  customer 
satisfaction  metrics  to  measure  the  initiative’s  level  of  success  in  the  areas  of  schedule, 
credibility,  and  performance.  The  second  recommendation  is  to  continue  to  shift  the 
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focus  of  acquisition  workforce  training  from  the  traditional  proeess  and  proeedures 
cookbook  method  of  doing  aequisitions  to  innovative,  business  oriented  training.  And  it 
is  reeommended  that  Defense  Finance  and  Aceounting  Serviee  and  Defense  Contract 
Management  Agency  civilians  be  included  in  Professional  Military  Education  classes. 

The  third  reeommendation  is  that  guiding  prineiples  be  created  by  each  functional  lead 
within  SAF/AQ  for  those  in  the  aequisition  eommunity  within  their  funetional  areas, 
using  those  established  by  SAF/AQC  as  a  model.  The  fourth  recommendation  is  to 
examine  the  waiver  proeess  and  make  it  more  user  friendly.  The  fifth  recommendation  is 
clarify  future  initiatives;  this  ean  be  done  by  making  the  initiatives  very  specific,  sending 
out  metries  and  a  commander’s  intent  message  with  the  initiatives  when  they  are  first  sent 
out,  and  not  using  buzzwords  (e.g.,  aequisition  reform).  The  sixth  recommendation  is  to 
create  a  database  of  success  stories  and  failures  at  implementing  the  “Foeus  on  results, 
not  process”  initiative.  The  seventh  recommendation  is  for  the  Aequisition  Center  of 
Exeellenee  (ACE)  offices  to  hold  regular  road  shows  discussing  the  initiative  and  others. 
The  eighth  reeommendation  is  to  ask  the  senior  Air  Foree  leadership  to  send  out  regular 
and  elear  statements  of  support  for  the  initiative  and  acquisition  reform  efforts.  Table  7  is 
a  summary  of  the  reeommendations  made  by  this  researeh. 


Table  7.  Summary  of  Researeh  Reeommendations 


Recommendations 

1. 

Customer  satisfaction  metrics  to  measure  suceess  of  initiative 

2. 

Innovation  training  for  aequisition  workforce 

3. 

Create  functional  area  guiding  principles 

4. 

Make  waiver  process  easier 

5. 

Clarify  future  initiatives 

6. 

Create  database  of  initiative  implementation  success  and  failure  stories 

7. 

ACE  offiee  road  shows 

8. 

Consistent  statement  of  initiative  support  from  senior  Air  Foree  leaders 
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Future  Research 


In  addition  to  drawing  conclusions  and  making  recommendations  based  on  the 
literature  and  data  analysis,  it  is  suggested  that  future  studies  continue  the  researeh  stream 
begun  by  this  projeet.  The  future  research  would  consist  of  a  quantitative  analysis  that 
would  validate  the  findings  of  the  qualitative  researeh. 

As  part  of  future  research,  a  questionnaire  ean  be  constructed  using  the  data 
gathered  from  the  interviews.  The  questionnaire  ean  be  used  to  evaluate  the  generated 
metrics  along  the  dimensions  of  “good”  metrics;  specific  metrics  that  apply  to  the 
eategories  of  metries  this  researeh  suggests  ean  then  be  identified  during  future  researeh. 
A  questionnaire  development  proeess  is  provided  below. 

Quantitative  Data  Analysis 

During  the  seeond  phase  of  the  researeh,  a  panel  of  experts  can  provide  an 
independent  assessment  of  whether  or  not  the  metries  suggested  for  the  Lighting  Bolt  of 
interest  possess  the  eharaeteristies  of  good  metries.  They  ean  use  a  questionnaire 
construeted  using  the  data  gathered  from  the  interviews.  The  questionnaire  can  be  used 
to  evaluate  the  generated  metries  along  the  dimensions  of  “good”  metries.  The  metries 
can  be  numerically  ranked  against  the  dimensions  of  good  metrics  using  a  seven-point 
interval  seale.  Univariate  statistics  can  then  be  used  to  analyze  the  eentral  tendeney, 
dispersion,  and  seore  distributions  of  the  resulting  data. 

Response  Format. 

A  seven  point  scale  can  be  used  for  the  quantitative  phase  because  seven  point 
seales  are  the  best  ehoice  in  terms  of  subjeets  being  able  to  elearly  distinguish  scale 
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values  from  one  another,  and  the  best  for  reliability  (Cox,  1980:414;  Sehwarz  and  others, 
1991:571).  The  rating  seale  for  this  researeh  also  eonsists  of  positive  numbers  beeause 
scales  being  used  to  assess  the  intensity  of  individual  attributes  should  be  formatted  from 
zero-to-positive  values;  this  stresses  that  the  question  being  asked  deals  with  the  presence 
or  absence  of  the  given  attribute,  instead  of  the  presence  of  its  opposite  (Schwarz  and 
others, 1991 :578).  It  can  be  determined  in  the  quantitative  phase  whether  the  survey 
instrument  will  use  rating  scales  tailored  to  fit  each  attribute  or  if  it  is  more  appropriate  to 
use  a  more  universal  scale  for  all  of  the  attributes.  These  findings  support  using  the 
measurement  scale  described  above. 

Validation. 

A  panel  of  experts  can  be  provided  with  a  list  of  all  the  metrics  suggested  by  the 
interview  data.  They  can  then  be  asked  to  refine  the  list  and  regroup  the  metrics  as 
necessary,  in  order  to  eliminate  redundancies.  The  panel  of  experts  can  then  be  asked  to 
quantitatively  assess  the  refined  list  of  metrics  against  the  attributes  of  good  metrics.  By 
completing  these  two  tasks,  the  panel  of  experts  can  provide  content  validity  for  the 
survey  instrument  developed.  The  survey  instrument  can  be  further  validated  in  a  follow- 
on  effort  that  collects  data  from  a  representative  sample,  analyzing  the  responses 
statistically. 

Measure  of  Central  Tendency. 

The  mean  rating  for  each  metric  can  be  calculated  across  all  the  participants. 

These  ratings  can  then  be  ranked  from  highest  to  lowest,  starting  with  the  most  important 
attribute.  Then  the  ratings  can  be  ranked  against  the  second  most  important,  and  so  on. 
The  hierarchy  of  attribute  importance  can  be  based  on  the  findings  of  the  qualitative 
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phase  of  the  research.  This  can  show  how  the  average  participant  rates  each  metric 
against  the  characteristics  of  good  metrics. 

But  because  the  mean  gives  more  weight  to  outliers  that  could  skew  the  data,  the 
median  score  for  each  metric  rating  can  also  be  determined  to  further  describe  the  central 
tendency  (Dooley,  2001:321). 

Measure  of  Dispersion. 

Next,  the  dispersion  of  the  data  can  be  analyzed  to  determine  how  much  the  data 
varies  above  or  below  the  mean.  In  this  context,  this  value  tells  how  much  the  ratings  of 
this  small  sample  differ  among  themselves  (Kachigan,  1991:41).  The  standard  deviation 
can  be  calculated  to  describe  the  data  dispersion  because  it  incorporates  the  distance  from 
the  mean  for  each  observation,  versus  using  the  range  which  only  uses  the  two  most 
extreme  observations  and  is  susceptible  to  skewness  by  outliers  (Dooley,  2001:321). 

If  participants’  ratings  vary  greatly  from  one  another,  it  may  be  an  indicator  that 
either  the  metrics  or  attributes  are  ill-defined,  or  it  could  mean  that  there  is  something 
unclear  about  the  format  of  the  questionnaire.  Conversely,  clustered  ratings  (i.e.,  ratings 
that  are  closely  grouped  together)  may  support  that  the  questionnaire  is  sound,  and  they 
may  provide  a  more  accurate  measurement  of  the  quality  of  the  metrics  against  the 
attributes. 

Score  Distributions. 

Then,  the  score  distributions  can  be  shown  through  rank-ordering  and  frequency 
counts.  Following  Nunnally’s  (1959:46)  suggestion  for  analyzing  data  when  the  number 
of  subjects  is  small,  the  survey  instrument  score  distributions  will  be  described  by  rank¬ 
ordering.  For  example,  if  a  metric  has  the  highest  score  out  of  five  metrics  for  the 
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dimension  of  simplicity,  it  can  be  easily  understood  how  simple  the  metric  is  in 
comparison  to  the  other  metrics.  After  the  central  tendency  and  dispersion  of  the  data  are 
analyzed,  the  metrics  can  be  ranked  to  illustrate  which  metrics  possess  the  highest 
number  of  good  attributes  in  comparison  to  one  another. 

And  for  a  broader  application,  the  frequency  of  how  many  times  each  metric  is 
rated  as  having  the  various  attributes  and  how  many  times  they  were  rated  as  not  having 
the  attributes  will  be  interpreted  from  the  data.  This  can  provide  a  more  generalized 
rating  of  ‘yes  -  a  particular  metric  is  simple,  or  is  meaningful  to  the  customer,  or  relates 
to  the  strategic  goals  of  the  organization,’  and  so  on,  or  a  generalized  rating  of  ‘no  -  it  is 
not  simple,’  etc.  Analysis  of  the  quantitative  data  can  provide  support  for  conclusions 
reached  through  the  qualitative  phase  of  the  research. 

Discussion 

This  project  was  designed  to  help  better  understand  the  effectiveness  of  Air  Force 
acquisition  initiatives,  using  the  Lightning  Bolt  initiatives  as  a  case  study,  and  to  establish 
acquisition  based  measures  of  success.  Specifically,  this  research  focused  on  the 
Lightning  Bolt  2002  initiative  that  is  most  closely  linked  with  the  goal  set  for  the  Air 
Force  to  improve  speed  and  credibility:  “Focus  on  results,  not  process”  (Senate  Armed 
Services  Committee,  2002).  This  research  examined  the  literature  to  provide  a  historical 
context  for  the  Lightning  Bolt  2002  acquisition  initiatives,  and  to  explain  the  importance 
and  composition  of  metrics.  From  the  literature  a  three  step  generic  process  for  creating 
metrics  and  the  core  attributes  of  good  metrics  were  identified.  Next,  the  research 
described  the  interview  process  used  to  collect  data  and  the  qualitative  method  used  to 
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analyze  the  data.  Then  the  researeh  explained  the  themes  that  evolved  from  the  data  and 
the  resulting  eonclusions.  Finally,  the  research  made  recommendations  that  could  be 
implemented  to  further  improve  the  performance  of  the  initiative  and  made  suggestions 
for  future  research  opportunities. 

This  study  gives  Air  Force  leaders  clear,  implementable  metrics  that  can  be  used 
as  measures  of  success  for  the  “Focus  on  results,  not  process”  initiative,  and  provides 
recommendations  that  can  be  used  to  improve  this  initiative’s  performance  and  that  of 
future  corporate  Air  Force  acquisition  initiatives.  This  study  also  gives  Air  Force  leaders 
insight  into  whether  or  not  this  initiative  and  others  like  it  are  an  appropriate  and  effective 
way  to  drive  the  changes  they  are  meant  to  bring  about.  Finally,  from  a  broader 
perspective,  the  framework  used  in  this  study  can  be  used  to  develop  measures  of  success 
for  other  corporate  level  Air  Force  acquisition  initiatives. 


85 


Appendix  A:  Interview  Assistance  Request  Letter 


23  August  2003 


Colonel  Rita  Jordan 
AFIT/EN 
2950  Hobson  Way 
WP  AFB,  OH  45433-7765 

Mrs.  Marty  Evans 

Director,  AE  Acquisition  Center  of  Excellence 
Assistant  Secretary  (Acquisition) 

SAE/ACE 

1060  Air  Force  Pentagon 
Washington,  DC  20330-1060 

Dear  Mrs.  Evans 

Thank  you  for  the  sponsorship  the  Acquisition  Center  of  Excellence  (ACE)  office 
is  providing  for  the  thesis  research  of  Capt  Carey  Petit.  Col  Ralph  DiCicco’s  help  has 
been  immeasurable  in  starting  her  research  down  a  path  that  will  prove  useful  to  the  Air 
Force  acquisition  community.  I  am  writing  to  you  to  ask  for  your  continued  assistance 
with  the  next  step  of  this  research. 

A  large  portion  of  Capt  Petit’s  research  requires  interviewing  system  program 
office  (SPO)  members  at  various  managerial  levels  who  have  had  experience  with  the 
2002  acquisition  Eightning  Bolt  initiative  “Focus  on  results,  not  process.”  Ideally, 
individuals  throughout  the  chain  of  command  at  selected  SPOs  within  all  of  the  Air  Force 
Product  Centers  and  Air  Eogistics  Centers  would  be  targeted  for  interviews.  The  target 
audience  would  ideally  consist  of:  40-50  people;  an  equal  number  of  people  to  be 
interviewed  at  each  of  the  Product  and  Eogistics  Centers;  individuals  who  hold  varying 
levels  of  managerial  responsibility  (e.g.,  people  ranging  from  program  managers,  to 
Development  System  Managers,  to  System  Program  Directors,  and  potentially  all  the 
way  to  Center  Commanders).  The  target  timeframe  for  the  interview  process  would 
ideally  be  to  identify  the  interview  subjects  by  30  Sep  03  and  complete  the  interviews  by 
31  Oct  03. 

I  would  like  to  ask  for  the  assistance  of  the  ACE  office  in  identifying  individuals 
within  this  target  audience  who  have  experience  with  the  “Focus  on  results,  not  process” 
initiative.  It  is  my  understanding  that  the  Center  level  ACE  offices  have  the  resources  to 
identify  such  individuals  at  their  respective  Centers.  Any  assistance  your  office  or  the 
Center  ACE  offices  can  offer  in  identifying  these  interview  subjects  by  name,  location, 
phone  number,  and  email  address  would  be  a  great  help.  For  your  convenience,  a  draft 
letter  inviting  SPO  members  to  participate  in  this  interview  process  is  attached  for  your 
signature.  Capt  Petit  would  like  to  send  the  letter  to  the  SPO  members  on  your  behalf. 
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once  the  Center  ACE  offiees  have  identified  them.  Your  eontinued  support  of  this 
researeh  effort  is  greatly  appreeiated. 


If  you  have  any  questions,  please  eontaet  Capt  Petit  by  phone  at  (937)  252-3164 
or  by  email  at  earey.petit@afit.edu.  Your  help  in  loeating  interview  subjeets  who  are 
already  familiar  with  the  initiative  will  greatly  faeilitate  the  interview  portion  of  this 
researeh.  Thank  you  for  your  help  with  this  matter. 

Sineerely 


RITA  A.  JORDAN,  Colonel,  USAF 
Assoeiate  Dean 

Graduate  Sehool  of  Engineering  and  Management 
(Thesis  Committee  Chair) 


CAREY  PETIT,  Captain,  USAF 

Graduate  Sehool  of  Engineering  and  Management 


Attaehment: 

Interview  Invitation  Fetter 
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Appendix  B:  Interview  Invitation  Letter 


3 1  October  2003 


1560  Wilson  Blvd. 

Suite  901 

Arlington,  VA  22209 

Potential  Interview  Subjeet 
1050  Air  Foree  Pentagon 
W ashington,  D.C.  20330-1050 

Dear  (Interview  Partieipant’s  Name) 

Through  the  assistanee  of  the  Center  ACE  offiees,  you  are  one  of  a  very  small 
group  who  have  been  identified  as  having  experienee  using  or  knowledge  of  the  2002 
aequisition  Lightning  Bolt  initiative  “Foeus  on  results,  not  proeess.”  As  your  sehedule 
permits,  Capt  Carey  Petit  would  like  to  set  up  an  appointment  to  interview  you  about 
your  experienees  with  this  aequisition  reform  initiative.  Capt  Petit  is  a  graduate  student 
at  the  Air  Foree  Institute  of  Teehnology.  Her  master’s  thesis  researeh  is  on  developing 
measures  of  sueeess  for  the  “Foeus  on  results,  not  proeess”  initiative. 

Capt  Petit’s  researeh  will  inelude  interviews  with  people  throughout  Air  Foree 
Materiel  Command  who  hold  positions  at  various  managerial  levels.  I  am  personally 
interested  in  what  you,  the  front-line  aequisition  praetitioners,  think  about  these 
initiatives  and  their  appropriateness.  Most  importantly,  this  will  give  me  the  neeessary 
insight  to  make  ehanges  and  improvements  that  help  us  all  serve  the  Air  Foree  better. 

Attaehed  is  a  eopy  of  the  interview  questions  that  will  be  asked.  However,  the 
interview  is  meant  to  be  only  partially  struetured  to  allow  for  the  free  flow  of  ideas  that 
are  outside  the  eonfines  of  the  preset  interview  questions  and  to  maximize  the  time 
available  for  open  diseussion.  Information  colleeted  during  the  interview  will  be 
eompletely  eonfidential;  no  individual  will  be  identified  in  any  way  in  Capt  Petit’s  thesis 
or  related  published  artieles.  The  interview  questions  have  been  approved  through  the 
Air  Foree  Personnel  Center  and  assigned  the  offieial  survey  number  of  USAF  SCN  03- 
098. 
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Thank  you  in  advance  for  your  willingness  to  assist  Capt  Petit  in  her  research 
efforts.  She  will  be  in  toueh  with  your  offiee  to  set  up  an  appointment  for  your  interview. 
If  you  have  any  questions,  Capt  Petit  ean  be  reaehed  by  phone  at  (937)  252-3164  and  by 
email  at  earey.petit@alit.edu. 


Sineerely 
//  SIGNED// 


MARTHA  T.  EVANS 
Director 

Acquisition  Center  of  Exeellenee 


Attaehment: 

Interview  Question  Eist 
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Attachment:  Interview  Question  List 


1 .  What  do  you  think  the  2002  Lightning  Bolt  “Focus  on  results,  not  process”  means? 

2.  What  do  you  think  are  the  results  it  is  trying  to  accomplish?  What  are  its  desired 
outcomes? 

3.  Do  you  think  these  goals  are  appropriate? 

4.  What  metrics  would  you  use  to  measure  those  results  or  outcomes? 

5.  What  do  you  consider  to  be  the  most  important  aspects  of  the  “Focus  on  results,  not 
process”  acquisition  reform  initiative? 

6.  What  should  be  the  next  step  in  “focusing  on  results”? 

7.  How  did  you  hear  about  the  “Focus  on  results,  not  process”  initiative? 

8.  How  is  your  organization  implementing  it? 

9.  What  has  been  your  role  in  the  development  and/or  implementation  of  this  initiative? 

10.  How  would  you  know  if  you  are  succeeding  at  implementing  the  initiative? 

1 1 .  What  kind  of  support  is  your  organization  getting  in  order  to  implement  the  initiative? 

12.  Are  you  getting  the  support  you  need  to  implement  the  initiative? 

13.  How  is  your  organization  supporting  you  in  your  attempts  to  implement  the  initiative? 

14.  Have  you  heard  of  any  success  stories  at  implementing  the  initiative? 

15.  Have  you  heard  of  any  big  failures  at  implementing  the  initiative? 

16.  What  do  you  think  it  takes  to  be  successful  in  implementing  any  acquisition  reform 
initiative? 

17.  What  is  important  to  you  in  an  acquisition  reform  initiative? 

18.  Is  there  anything  else  you  would  like  to  discuss? 

19.  Do  you  have  any  questions  for  me? 
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Appendix  C:  Interview  Related  Exchanges 


23  August  2003 


Captain  Carey  Petit 
AMC  P.O.  Box  33768 
WP  AFB,  OH  45433 

Potential  Interview  Subject 

System  Program  Director,  XXX  Platform 

XXXXXAFB,XX  00000 

Dear  (Interview  Subject’s  Name) 

I  want  to  thank  you  again  for  agreeing  to  assist  me  with  my  effort  to  develop 
measures  of  success  for  the  2002  acquisition  Lightning  Bolt  initiative  “Focus  on  results, 
not  process.”  I  appreciate  you  taking  time  out  of  your  busy  schedule  to  participate  in  the 
interview  we  have  scheduled  for  23  Sep  03  at  1300hrs.  Please  let  me  know  if  any 
conflicts  arise  with  the  current  time  we  have  scheduled  for  our  interview.  I  look  forward 
to  talking  with  you. 

I  can  be  reached  by  phone  at  (937)  252-3164  and  by  email  at  carey.petit@alit.edu, 
if  you  have  any  questions. 


Sincerely 


CAREY  PETIT,  Capt,  USAF 
AEIT  Graduate  Student 
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Appendix  C:  Interview  Related  Exchanges 
Interview  Script 

NOTE:  Have  interview  subject  fill  out  checklist  below  at  the  beginning  of  the  interview 
during  the  set-up  time  (i.e.,  while  tape  recorder  is  being  set  up  and  other  preparations  are 
being  made): 

Demographics 

Age  and  Gender? 

Occupation 

-  Functional  area? 

Organization 

-  Number  of  organizational  levels  that  separate  you  from  the  Center 
Commander? 

Supervisor 

-  If  so,  how  many  people  do  you  supervise?  What  are  their  occupations? 

Tenure 

-  Time  in  service? 

Introduction 

Good  morning  (afternoon),  my  name  is  Capt  Carey  Petit.  I  am  a  student  in  the 
Strategic  Purchasing  Master’s  Degree  Program  at  the  Air  Force  Institute  of  Technology 
(AFIT).  The  information  I  collect  through  this  interview  will  be  part  of  my  master’s 
thesis.  My  thesis  is  about  developing  measures  of  success  for  the  2002  Fighting  Bolt 
initiative  “Focus  on  results,  not  process.”  I  want  to  thank  you  for  taking  time  out  of  your 
busy  schedule  to  answer  a  few  questions  for  me.  Your  help  will  potentially  assist  the  Air 
Force  Acquisition  Center  of  Excellence  (ACE)  in  establishing  metrics  for  this  initiative 
and  in  determining  if  such  initiatives  should  be  continued  in  the  future,  and  you  will 
greatly  assist  me  in  furthering  my  thesis  work. 

Confidentiality 

Fet  me  begin  by  saying  that  everything  you  say  today  will  be  completely 
confidential.  Your  comments  will  only  be  seen  by  me,  my  advisor,  and  a  transcriber. 
Only  general  feedback  on  the  responses  of  interview  subjects  will  be  used  in  my  thesis. 
No  individual  will  be  identified  in  any  way.  Any  quotations  that  are  used  in  my  final 
paper  will  be  altered  in  a  way  to  conceal  your  identity. 

The  interviews  conducted  during  my  research  will  be  analyzed  for  common 
themes.  These  common  themes  will  then  be  used  to  help  me  write  a  questionnaire  that 
will  assess  the  effectiveness  of  the  “Focus  on  results,  not  process”  initiative 
implementation  and  the  appropriateness  of  metrics  gathered  from  the  interview  data  as 
future  measurements  of  success  for  the  same  initiative.  The  questionnaire  will  potentially 
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be  filled  out  by  many  people  throughout  Air  Foree  Materiel  Command  (AFMC)  and  Air 
Foree  Spaee  Command  (AFSPC)  in  future  researeh  efforts. 

As  I  will  not  be  able  to  remember  everything  that  is  said  during  the  interview,  I 
will  be  taking  notes  and  would  like  to  ask  your  permission  to  record  our  conversation, 
which  will  then  later  be  transcribed.  No  one  other  than  me,  my  advisor,  and  a  transcriber 
will  read  my  notes  or  hear,  or  read  the  transcription  of,  the  interview.  At  any  time,  you 
can  read  my  notes  and  correct  any  mistakes  you  think  I  have  made.  And  if  at  anytime, 
you  would  like  to  stop  recording  for  any  reason,  please  let  me  know.  If  you  would  like,  I 
will  be  glad  to  forward  a  copy  of  this  interview  to  you  after  it  is  transcribed. 

Interview  Format 

I  apologize  up  front  for  watching  the  clock,  because  I  do  not  want  to  take  up  too 
much  of  your  time.  I  have  tried  to  limit  my  questions  so  as  not  to  exceed  30-45  minutes. 
However,  I  want  to  stress  that  the  interview  is  meant  to  be  somewhat  unstructured  and 
free-flowing.  So  if  there  is  anything  that  you  would  like  to  discuss  further  please  let  me 
know.  Do  you  have  any  questions  before  we  start? 

Interview  Items 

To  give  you  a  brief  summary  of  the  focus  of  my  research. . .  The  President, 
Secretary  Rumsfeld,  and  other  senior  Air  Force  leaders  are  continuing  to  seek  ways  to 
improve  the  Air  Force’s  acquisition  processes  through  various  means  of  acquisition 
reform.  A  series  of  acquisition  reform  initiatives  called  “Lightning  Bolts”  were  started  in 
1995  by  Ms.  Darleen  Druyun,  then  Principal  Deputy  Assistant  Secretary  of  the  Air  Force 
for  Acquisition  Management.  My  research  will  focus  on  the  2002  Lightning  Bolts 
entitled  “Focus  on  results,  not  process.”  This  initiative  drives  a  “clean-sheet”  approach  to 
acquisitions  by  streamlining  processes  in  order  to  remove  non  value-added  steps.  One 
recent  example  of  this  initiative  in  action  is  the  rewriting  of  the  regulations  that  govern 
weapon  system  procurement  within  the  Department  of  Defense,  the  DoD  5000  series; 
portions  of  the  regulation  were  eliminated  while  others  were  rewritten. 

Now,  from  your  personal  experiences,  I  would  like  you  to  think  of  acquisition  reform 
initiatives,  specifically  the  2002  Lightning  Bolt  initiative  “Focus  on  results,  not  process.” 
Think  of  your  role  in  this  reform  effort.  Also,  try  to  recall  the  activities  that  surrounded 
the  initiative  and  of  your  impressions  of  its  facilitation. 

«  Pause  a  moment  » 

OK,  let’s  get  started. . . 

1 .  What  do  you  think  the  2002  Lightning  Bolt  “Focus  on  results,  not  process”  means? 

2.  What  do  you  think  are  the  results  it  is  trying  to  accomplish?  What  are  its  desired 
outcomes? 
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3.  Do  you  think  these  goals  are  appropriate? 

4.  What  metries  would  you  use  to  measure  those  results  or  outeomes? 

5.  What  do  you  consider  to  be  the  most  important  aspects  of  the  “Focus  on  results,  not 
process”  acquisition  reform  initiative? 

6.  What  should  be  the  next  step  in  “focusing  on  results”? 

7.  How  did  you  hear  about  the  “Focus  on  results,  not  process”  initiative? 

8.  How  is  your  organization  implementing  it? 

9.  What  has  been  your  role  in  the  development  and/or  implementation  of  this  initiative? 

10.  How  would  you  know  if  you  are  succeeding  at  implementing  the  initiative? 

1 1 .  What  kind  of  support  is  your  organization  getting  in  order  to  implement  the  initiative? 

12.  Are  you  getting  the  support  you  need  to  implement  the  initiative? 

13.  How  is  your  organization  supporting  you  in  your  attempts  to  implement  the  initiative? 

14.  Have  you  heard  of  any  success  stories  at  implementing  the  initiative? 

15.  Have  you  heard  of  any  big  failures  at  implementing  the  initiative? 

16.  What  do  you  think  it  takes  to  be  successful  in  implementing  any  acquisition  reform 
initiative? 

17.  What  is  important  to  you  in  an  acquisition  reform  initiative? 

18.  Is  there  anything  else  you  would  like  to  discuss? 

19.  Do  you  have  any  questions  for  me? 
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Appendix  D:  Summary  of  Interview  Themes  and  Participant  Responses 


When  reviewing  Appendix  D,  be  aware  of  the  following  factors:  a)  more  than 
one  theme  could  apply  to  a  participant’s  response  for  any  given  interview  question  (e.g., 
Table  4  in  chapter  four  shows  how  one  response  from  a  participant  applied  to  the  themes 
of  schedule,  cost,  and  performance);  and  b)  the  percentage  of  participants  among  whom 
the  themes  occurred  may  seem  elevated  due  to  the  low  number  of  participants  who 
participated  in  the  research  (e.g.,  one  participant  equals  four  percent  of  the  total 
participants). 
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Interview  Question:  What  do  vou  consider  to  be  the  most  important  asoects  of  the  "Focus  on  results,  not  z>rocess"  acauisition  1 

reform  initiative? 
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Themes  from  Qualitative  Analysis  of  Interview  D ata 

Support  the  Agile  Acquisition  strategy;  provide  capability  in  a  timely  way  without 
getting  bogged  down  in  the  processes 

Change  people’s  way  of  thinking,  be  creative,  innovative,  and  use  common  sense  I 

Get  people  to  think  about  the  outcome  not  the  how 

Promises  made,  promises  kept  | 

Roadblocks  exist  to  accomplishing  initiative  outcomes  from  1)  middle 
management,  2)  SAF/AQ  staff  and  other  services,  and  3)  contracting 

Unchain  the  process  and  make  bureaucrats  look  at  the  big  picture  | 

Goals  are  unclear  | 

Challenge  decision  makers  when  necessary,  risk  management  | 

Freeing  up  the  workforce,  empowering  them  | 

Interview  Question:  Do  vou  think  these  soals  are  az>z>rovriate?  1 

Themes  from  Qualitative  Analysis  of  Interview  D ata 

Positive  reply 

Negative  reply  or  criticism  | 

Themes  from  Qualitative  Analysis  of  Interview  Data 

Organizational  culture 

Entrepreneurial  mindset  | 

Responsiveness  to  customer  | 

C  ommunic  ation  | 

Interview  Question:  How  did  vou  hear  about  the  "Focus  on  results,  not  vrocess"  i?-. 

Themes  from  Qualitative  Analysis  of  Interview  Data 

Did  not  hear  about  initiative  until  this  research  project  I 

Through  chain  of  command  and  normal  information  channels  | 

Direct  involvement  with  Dru3mn  (initiative  originator)  I 

Helped  write  the  initiative 

Acquisition  reform  training  | 

Sambur's  Agile  Acquisition  policy  letter  | 

Working  in  Acquisition  Center  of  Excellence  (ACE)  office 
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Interview  Question:  What  is  the  next  step  in  "focusins.  on  results"? 

Themes  from  Qualitative  Analysis  of  InteiAiewData 

Change  the  acquisition  workforce  culture 

Apply  the  initiative  to  the  processes  that  support  Evolutionary  Acquisition 

Training  and  education 

Expectation  management  with  the  warfighters  and  Air  Staff,  and  in  turn  Congress 

Stop  smacking  people’s  hands  for  doing  acquisitions  differently  than  it  has  always 
been  done 

Corporate  buy-in  across  oversight  organizations  within  the  Air  Force  and  the 
Department  of  Defense 

Senior  Air  Force  leadership  buy-in 

Provide  examples  of  specific  success  stories 

Roadshows 

Be  specific  about  what  constitutes  “results” 

Move  people  fi'om  oversight  to  execution  roles 

Take  people  out  of  the  approval  chain  who  are  not  in  the  decision  chain 

Bust  roadblocks 

Enprove  filtering  down  of  initiatives;  initiatives  lose  a  lot  of  punch  by  the  time  they 
get  to  the  SPO  level  workers 

Release  a  new  set  of  initiatives  on  a  more  practical  and  specific  level 

Obtain  feedback  fi'om  the  troops 

Look  at  how  we  make  acquisitions  work  in  the  future 

Eiclude  demonstration  of  initiative  implementation  as  part  of  appraisals 

Send  out  messages  about  when  programs  and  people  failed,  but  were  still 
rewarded  for  trying  and  being  innovative 

Set  a  standard  or  cut-off  point  where  failing  programs  are  turned  off 

Get  people  willing  to  take  risks 

Obtain  buy-in  firom  functional  and  operational  communities 

Follow  through 
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Interview  Question;  How  is  your  orsanization  implementing^  it? 

Themes  from  Qualitative  Analysis  of  Intei"view  Data 

ACE  oflSces  assist  programs  to  challenge  burdensome  processes  and  try  to 
influence  people  to  use  the  philosophy  of  the  initiative 

Stress  full  participation  of  Integrated  Product  Teams  (TPTs)  and  ensure  IPTs 
include  the  warfighters,  contractors,  and  contracting  officers 

Rewrote  Air  Force  Instructions  and  Air  Force  Federal  Acquisition  Regulation 
Supplements  and  other  guiding  documents  to  fi'ee  up  people 

Negative  replies  about  not  using  the  initiative 

Opportunity  management;  being  proactive  versus  reactive,  being  creative,  and 
using  risk  management 

Challenged  SPO  members  to  use  common  sense,  and  then  if  it  is  not  the  law  and 
does  not  make  sense  to  break  it 

Ensured  Requests  for  Proposals  and  proposals  are  outcome  based 

Developed  and  implemented  training  on  the  using  the  initiative 

Applied  initiative  as  appropriate  to  SPO  programs 

Used  internal  teamwork  to  change  SPO  mindset  firom  no  unless  you  can  prove 
yes,  to  yes  until  a  roadblock  is  found  that  says  it  cannot  be  done 

Empowered  SPO  program  leads,  lessened  emphasis  on  functional  leads;  remove 
non- value  added  parts  of  the  chain  of  command 

new  Question:  What  has  been  your  role  in  the  development  and/or  implement 

Theme. s  from  Qualitative  Aiialysi.s  of  InteiAiew  Data 

Being  an  enabler  for  your  team 

Developer  or  author  of  the  initiative 

Endorser  and  advocate  of  the  initiative 

Had  no  role  in  the  development  of  the  initiative 

Provide  advice  to  senior  leadership  on  ways  to  implement  the  initiative 

Helped  rewrite  policies  as  a  result  of  using  the  initiative 

Provided  training  for  SPOs  on  how  to  use  the  initiative 

Managed  customer  expectations 

Architect  for  implementation  of  the  initiative  within  a  SPO 
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Themes  from  Qrralitative  Arralysis  of  Literxiew  Data 

Top  down  support 

ACE  help  in  planning  for  program  events 

Contractor  support 

Initiative  training;  risk  management  training  and  Discovery  Map  training 

Being  left  alone  and  trusted  to  go  implement  the  initiative  is  the  best  support 

Internal  support 

Congressional  language  can  be  a  big  supporter 

No  support  IS  being  given 

The  bureaucracy  is  fitting  implementation  of  the  initiative 

Senior  leaders  empowered  the  workforce  to  go  out  and  implement  the  initiative, 
but  they  are  not  preaching  it  enough  themselves,  need  strong,  consistent  advocacy 

Headquarters  puts  the  initiatives  out  but  does  not  have  to  live  with  them 

hew  Question:  Are  you  setting,  the  support  you  need  to  implement  the  initiate 

Themes  from  Qualitative  Analysis  of  InteixiewData 
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Not  getting  support  from  related  processes  and  people  outside  their  control 

Sort  of  or  do  not  know 

Themes  from  Qrralitative  Analysis  of  Interview  Data 

Support  and  encouragement  from  leadership  within  their  direct  chain  of  command 

By  being  trusted  to  do  the  job  and  being  left  alone  to  do  it 

Strong  Acquisition  Center  of  Excellence  (ACE)  support 

No  kick-backs  from  System  Program  Office  (SPO)  level  workers  means  they 
have  accepted  the  challenge  to  implement  the  initiative 

Not  enough  engaging  from  Senior  Executive  Service  or  General  Officer  level 
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No  specific  examples 

Provided  examples 

Process  level  success  stories 

1  Addition  of  sotjrce  selection  plan  to  System  Acquisition  Management  Plan  I 

Using  same  pricing  structure  for  Proposal  Analysis  Report  and  Price 
Competition  Memo 

Elimination  of  Mission  Need  Statement  and  Operational  Requirements 
Document  for  desktop  computer  purchase 

Reduction  in  training  approval  time 

Delegation  of  contract  approval  authority  so  people  rarely  have  to  go 
through  contracting  headquarters 

1  Weapon  system  program  success  stories  I 

Crystal  Modification  Program 

Global  Havs^k 

1  Micro  ImpTolse  Radar  I 

Patient  Support  Pallet 

Weaponization  of  Predator 

ChemS entry  Chemical  Detectors 

Interview  Question:  Have  vou  heard  of  any  bis  failures  at  imz>lementins  the  initiati 

Theme. s  from  Qualitative  Analy.si.s  of  Interview  Data 

No  specific  examples 

Provided  examples 

Program  level  success  stories 

ChemS  entry  Chemical  Detectors 

I  F/A-22  Aircrafl:  program  I 

Program  Executive  OfiScer  (PEO)  shop  move  to  Air  Force  Materiel 
Command  Product  Centers  requiring  more  manpower  than  when  they 
were  at  Pentagon 

Process  level  success  stories 

Funding  reprogramming  taking  one  hundred  and  thirty  days  just  to  leave 
the  Pentagon 

Electronic  Systems  Center  System  Program  Office  not  using  zero -baseline 
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stability,  eKpectations  management) 
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